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PREFACE 


The  National  Conference  on  Mental  Health  Sta- 
tistics meets  annually  under  the  sponsorship  of  the 
Biometry  Branch  of  the  National  Institute  of  Mental 
Health.  The  purpose  of  these  meetings  is  to  provide  a 
forum  for  discussion  of  statistical  activities  in  local, 
State,  regional,  and  national  mental  health  programs. 
Participants  in  the  meetings  include  not  only  statisti- 
cians from  these  mental  health  programs  but  also  pro- 
gram administrators  and  clinicians  who  are  concerned 
with  the  development  and  use  of  statistical  data  in  pro- 
gram planning,  control,  and  evaluation,  and  in  patient 
care  and  management. 

The  proceedings  of  the  conferences  are  of  interest 
not  only  to  those  who  participate  in  the  meetings,  but 
also  to  many  other  mental  health  program  professionals 
who  are  not  able  to  be  present.  This  publication,  then, 
is  for  the  purpose  of  disseminating  and  making  more 
widely  available  the  results  of  the  conferences;  and  it 
also  supplies  a useful  record  and  report  of  the  meetings’ 
substantive  information  content. 


The  programs  for  the  annual  meetings  of  the  con- 
ference are  planned  by  an  advisory  committee  of  State 
representatives  and  NIMH  Regional  Office  and  Cen- 
tral Office  staff.  The  meeting  time  is  devoted  to  the 
consideration  of  current  problems  relating  to  data  col- 
lection, processing,  analysis,  presentation,  and  use  in 
mental  health  program  management  and  in  biosta- 
tistical  and  epidemiological  research.  Twelve  papers 
from  the  1967,  1968,  and  1969  meetings  have  been 
selected  for  publication  in  this  report.  Other  papers, 
equally  as  important,  were  presented  at  these  meetings 
but  are  not  included  in  this  report  for  various  reasons, 
such  as  the  reference  to  large  numbers  of  illustrations; 
publication  in  other  sources,  etc. 

The  staff  of  the  Biometry  Branch  is  indebted  to  each 
of  the  conference  speakers  and  to  members  of  the 
advisory  committees  who  worked  on  the  difficult  task 
of  designing  timely  and  interesting  programs  on  the 
conference  topic,  “Statistics  in  Mental  Health 
Programs.” 

Cecil  R.  Wurster. 
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COMMUNITY  MENTAL  HEALTH  STATISTICAL  PROGRAMS 

OF  THE  FUTURE 

Bernard  L.  Bloom,  Ph.  D. 

University  of  Colorado 


Many  of  you,  I am  sure,  are  familiar  with  the  chal- 
lenging vision  of  tomorrow’s  mental  health  statistician 
enunciated  by  Morton  Kramer  in  his  1966  paper  en- 
titled “Mental  Health  Statistics  of  the  Future.”  While 
I must  confess  some  initial  surprise  in  seeing  the  paper 
published  in  Eugenics  Quarterly,  in  a way  it  seems 
altogether  fitting.  Eugenics  is  the  science  of  improving 
the  qualities  of  the  human  race,  and  it  may  very  well 
be  that  the  mental  health  statistician  will  contribute 
to  that  goal,  just  as  the  vital  statistician  has  helped  in 
the  battles  for  reduced  morbidity  and  mortality  from 
the  nutritional  and  infectious  diseases  and  for  the 
extension  of  life  expectancy. 

The  Kramer  paper  concerns  itself  with  the  nature 
and  content  of  mental  health  statistics  of  the  future  and 
is  written  from  the  point  of  view  of  a “Federal  pro- 
gram, located  in  the  National  Institute  of  Mental 
Health,  responsible  for  the  collection  and  analysis  of 
data  on  the  mental  health  of  the  Nation.”  (Kramer, 
1966,  p.  186.)  What  I would  like  to  do  this  morning 
is  to  discuss  with  you  some  notions  about  the  roles 
of  mental  health  statisticians  of  the  future  “in  the  best 
of  all  possible  worlds,”  from  the  point  of  view  of  the 
local  community  rather  than  from  the  national  level. 

At  the  outset  it  is  important  to  note  that  statistical 
programs  must  have  some  specific  objectives  of  their 
own — they  can  rarely  satisfy  everyone.  Local  mental 
health  program  directors  have  the  problem  of  exam- 
ining national  statistical  reports  and  trying  to  make 
use  of  them  in  terms  of  their  programs.  National  sta- 
tistics, so  crucial  for  presenting  the  national  mental 
health  goals  and  problems  to  the  U.S.  Congress,  seems 
sterile  and  far  removed  from  the  urgent  problems  in 
the  local  community.  Interstate  differences  in  mental 
health  statistics  are  very  difficult  to  interpret.  Admis- 
sion rates,  readmission  rates,  average  length  of  hospital- 
ization, staff-patient  ratio,  cost  per  patient  per  day, 
even  figures  specific  for  age  or  sex  or  diagnosis  seem 
irrelevant  to  local  program  directors  who  are  interested 
in  improving  quality  of  services.  For  these  figures  tell 


us  little  about  treatment  effectiveness,  or  the  mental 
health  of  the  community,  or  the  local  sources  of  psycho- 
pathology, or  what  happens  to  patients  as  they  try  to 
reintegrate  themselves  into  the  community  structure. 
We  can  similarly  expect  that  statistical  programs  organ- 
ized to  be  directly  responsive  to  local  goals  and  prob- 
lems will  be  difficult  to  utilize  at  the  national  level. 
National  statistics  must  rely  on  nationally  agreed  upon 
definitions  and  categories.  There  would  be  no  way  to 
summarize  and  present  national  trends  if  the  source 
data  consisted  of  a series  of  statistical  reports  uniquely 
tailored  to  unique  communities.  But  since  there  are  so 
many  here  who  speak  eloquently  about  the  national 
interest,  I believe  I can  play  a useful  role  by  focusing 
on  local  mental  health  programs.  Obviously,  the  inter- 
ests of  local  and  national  mental  health  programs  coin- 
cide at  many  points,  and  I would  hope,  therefore,  that 
these  remarks  will  have  some  general  relevance. 

It  is  well  to  remind  ourselves  that  what  is  happening 
in  mental  health  program  philosophy  in  the  United 
States  is  part  of  an  enormous  social  awakening.  The 
failure  of  the  American  mental  health  establishment 
to  deliver  effective  services  where  and  when  needed 
has  been  recognized.  Indeed,  the  failure  of  many  of 
our  general  health  and  welfare  programs  has  been 
acknowledged.  With  these  admissions  has  come  a pro- 
found rise  in  the  social  responsibility  of  our  attitudes 
toward  health,  education,  and  welfare,  a significant 
move  toward  the  integration  of  all  health  sendees,  and 
specifically  a growing  sense  of  urgency  regarding  im- 
proved mental  health  services  (see  Bloom,  1967). 

One  of  the  calls  to  arms  has  been  the  word  “com- 
prehensive.” The  term  is  often  not  clearly  defined. 
What  I want  to  do  is  try  to  show  how  a definition  of 
this  term  can  set  the  stage  for  looking  at  the  tasks  which 
face  the  community-based  mental  health  statistician. 
A comprehensive  mental  health  program  must  have  at 
least  six  characteristics.  First,  it  must  be  focused  on  the 
restoration  and  maintenance  of  mental  health  within 
the  limits  imposed  by  the  available  knowledge  at  our 
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disposal.  Second,  it  must  concern  itself  with  prevention 
in  the  widest  sense  of  that  term.  It  must  prevent  the 
occurrence  of  all  preventable  mental  disorders;  it  must 
avoid  prolongation  of  emotional  disorders  by  providing 
prompt  and  effective  treatment;  and  it  must  reduce 
the  frequency  and  severity  of  disability  associated  with 
emotional  disorders.  Third,  it  must  provide  continuous 
service,  in  health  as  well  as  in  sickness,  and  at  all  stages 
of  the  disorder  rather  than  only  at  its  most  severe. 
Fourth,  it  must  be  person-centered  rather  than  organ- 
centered,  paying  attention  to  the  interdependence  of 
mind  and  body,  to  the  interrelationships  of  environ- 
mental factors  with  interpersonal  factors  and  to  the 
behavior  of  the  individual  in  health  and  when  under 
stress.  Fifth,  the  program  must  concern  itself  with 
sufficiency  of  services,  providing  that  amount  of  serv- 
ice of  all  kinds  necessary  for  the  population  to  be 
served,  and  served  promptly,  and  sixth,  the  program 
must  emphasize  quality  of  service  as  well  as  quantity 
of  service.  I’d  like  to  comment  now  on  certain  aspects 
of  this  definition  in  terms  of  mental  health  statistics. 

If  the  statistician  is  to  help  develop  a comprehen- 
sive mental  health  program,  he  must  concern  himself 
with  extending  the  limits  of  knowledge.  It  is  no  longer 
sufficient  for  him  to  spend  all  his  energies  in  collecting 
and  reporting  on  program  activities  of  last  month, 
or  last  year,  or  of  2 or  3 years  ago,  depending  on  how 
far  behind  he  is.  He  must  also  be  in  front  of  mental 
health  programs.  A month  or  two  ago  I had  occasion 
to  review  the  statistical  methodology  associated  with 
developing  an  abridged  life  table.  I got  bogged  down 
at  one  point  and  had  to  yell  for  help.  One  of  the  places 
I called  was  the  Department  of  Preventive  Medicine  at 
the  University  of  Colorado  Medical  School.  I vaguely 
remembered  reading  an  announcement  that  a biostat- 
istician had  been  added  to  their  staff.  When  I finally 
tracked  him  down  and  told  him  my  problem,  he  re- 
plied, “I’m  not  that  kind  of  biostatistician.”  What  he 
meant,  it  turned  out,  was  that  he  was  a consultant  to 
the  medical  school  at  large,  helping  them  in  the  de- 
velopment of  research  studies  and  in  the  strategies  for 
the  analysis  of  data.  Community-based  mental  health 
programs,  it  seems  to  me,  need  statisticians  who  are 
also  “not  that  kind  of”  mental  health  statistician.  Our 
programs  need  people  who  can  combine  the  skills 
associated  with  program  monitoring  with  the  skills 
associated  with  both  basic  and  applied  research  de- 
signed to  extend  the  limits  of  our  knowledge. 

If  the  statistician  is  to  help  develop  a comprehen- 
sive mental  health  program,  he  must  be  alert  to  the 
whole  range  of  the  program,  to  its  full  range  of  pre- 
ventive activities.  In  this  context  there  is  an  absolutely 


exhilerating  range  of  challenging  activities.  In  the 
field  of  primary  prevention,  the  mental  health  statis- 
tician can  engage  in  two  related  activities.  First,  he 
can  help  identify  groups  of  people  who  are  unusually 
vulnerable  to  the  development  of  emotional  disorders 
for  which  community  services  would  be  desirable.  He 
needs  to  study  the  extent  to  which  emotional  disorders 
are  associated  with  other  kinds  of  social  disequilibrium, 
or  with  normative  crises  such  as  retirement,  pregnancy, 
or  family  deaths.  He  needs  to  ascertain  whether  there 
are  significant  mental  health  consequences  associated 
with  divorce,  suicide  attempts,  loss  of  employment, 
community  disasters,  accidents,  or  school  dropouts.  He 
needs  to  be  alert  to  the  extent  to  which  emotional  dis- 
orders are  unusually  common  among  neglected  and 
battered  children,  among  the  children  of  psychotic 
parents,  among  those  who  suffer  from  chronic  ill- 
nesses, among  the  culturally  deprived  and  the  racially 
or  religiously  discriminated,  among  those  who  come 
in  contact  with  the  courts,  with  welfare  agencies,  and 
with  health  facilities.  In  addition,  the  ecology  of  emo- 
tional disorders  needs  to  be  investigated.  What  are  the 
characteristics  of  the  neighborhoods  from  which  come 
our  patients?  In  this  connection,  census  tract  informa- 
tion is  invaluable  and  later  on  this  morning  I would 
like  to  give  you  an  example  of  a study  of  this  kind  (see 
Bloom,  1966.)  In  short,  the  mental  health  statistician 
needs  to  have  his  fingers  squarely  on  the  pulse  of  the 
clientele,  searching  clinical  case  folders  as  well  as  tabu- 
lated identification  data  for  possible  sources  of  psycho- 
pathology in  the  community. 

Second,  having  established  possible  foci  of  psychiatric 
description,  the  mental  health  statistician  needs  to  help, 
and  perhaps  take  the  leadership  in  the  development 
of  a series  of  alerting  mechanisms.  If  it  can  be  shown 
that  a parental  death  is  significantly  disruptive  to  chil- 
dren of  school  age,  for  example,  then  an  automatic 
alerting  system  should  be  established  which  would 
identify  such  children  when  a death  occurs.  In  this 
way  a preventive  intervention  service  can  be  provided 
through  the  schools.  If  emotional  disorders  are  un- 
usually common  within  a month  or  so  after  retirement, 
an  alerting  system  in  cooperation  with  local  employers 
can  trigger  an  anticipatory  guidance  program  or  can 
provide  consultation  for  the  establishment  of  such  a 
program.  One  recent  special  study  from  the  Biometry 
Branch  has  shown  the  increased  risk  of  psychiatric  hos- 
pitalization of  women  with  four  or  more  children  in 
contrast  with  those  with  smaller  families.  Here  is  an- 
other high-risk  group  which  can  be  easily  identified  for 
whom  special  preventive  services  can  be  designed — 
perhaps  nothing  more  complicated  than  an  annual  2- 
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week  vacation  away  from  the  children.  The  mental 
health  statistician  can  help  establish  a flagging  system 
to  make  this  possible. 

Regarding  secondary  prevention  services,  that  is,  the 
prompt  identification  of  cases  of  emotional  disorder 
and  the  prompt  provision  of  appropriate  treatment,  the 
community-based  mental  health  statistician  can  collect 
several  pieces  of  most  useful  information.  First,  by 
what  pathways  do  patients  arrive  at  the  service  facility? 
Can  the  traverse  of  his  network  be  made  more  efficient 
or  more  rapid?  Second,  how  long  is  it  between  the 
felt  need  for  service  and  the  petition  for  service?  Can 
this  time  interval  be  reduced  by  some  form  of  mental 
health  education,  for  example?  Third,  how  long  is  it 
between  the  petition  for  service  and  the  provision  for 
service?  Can  the  waiting  list  be  reduced?  Can  triage 
and  emergency  service  programs  be  introduced  to  re- 
duce the  waiting  time?  Fourth,  what  about  patients 
in  need  of  help  who  never  petition  for  help?  Can 
improved  case  finding  services  be  inaugurated? 

In  the  area  of  psychiatric  rehabilitation  or  tertiary 
prevention,  the  mental  health  statistician  can  study  the 
flow  into  the  community  of  patients  discharged  from 
State  mental  hospitals  and  other  extra-community  fa- 
cilities. What  followup  services  need  to  be  provided? 
With  what  frequency  is  it  found  that  failure  to  provide 
suitable  rehabilitative  services  results  in  a failure  to 
make  a satisfactory  adjustment  to  the  community?  Just 
as  a flagging  system  can  be  established  for  primary  pre- 
vention services,  so  a community  mental  health  center 
can  be  alerted  to  the  return  to  the  community  of  a psy- 
chiatric pat:ent  treated  elsewhere.  Regular  reports  to 
the  clinical  staff  of  the  degree  of  their  success  in 
promptly  reaching  patients  in  need  of  all  forms  of 
service  can  be  a most  useful  kind  of  document,  and  the 
mental  health  statistician  can  take  major  responsibility 
for  the  preparation  of  this  report. 

If  the  mental  health  statistician  is  to  help  develop  a 
comprehensive  community-based  program,  he  must 
be  concerned  about  continuity  of  care.  This  implies 
a kind  of  record  system  which  can  be  sensitive  to  fail- 
ures in  the  caretaking  network  to  provide  service  on 
demand,  to  missing  components  in  caretaking  network 
and  to  unnecessary  losses  of  patients  within  the  net- 
work. We  are  talking  here  about  a conscientious  pro- 
gram of  human  accountability.  We  know  from  studies 
conducted  in  the  Boston  area  that  half  of  the  patients 
referred  from  one  facility  to  another  never  get  to  the 
other  facility,  and  half  of  those  who  get  there  do  not 
get  definitive  service.  These  kinds  of  slippages  in  a 
mental  health  program  system  exist  in  part  because 
we  are  not  made  adequately  and  forcefully  aware  of 


them.  As  an  accountant  of  the  continuity  of  services 
provided  to  our  clientele,  the  mental  health  statisti- 
cian can  be  in  a key  position.  And  as  a diagnostician 
of  the  failures  in  continuity  of  service,  he  can  be  even 
more  useful.  What  proportion  of  patients  referred 
from  a particular  facility  to  a particular  facility  get 
there?  What  reasons  appear  to  account  for  the  failure 
to  arrive?  Could  special  provisions  be  instituted  to 
increase  the  arrival  rate?  For  those  patients  who  arrive 
at  the  facility  to  which  they  are  referred  what  was 
the  disposition?  If  the  facility  could  not  provide  serv- 
ice, was  a secondary  referral  made?  With  what  suc- 
cess? Could  a preliminary  telephone  call  by  the  origi- 
nal referral  source  have  increased  the  likelihood  that 
the  patient  would  receive  service  when  he  arrived? 
Answers  to  these  questions,  among  others,  can  be  pro- 
vided and  can  help  program  directors  in  developing 
a group  of  services  in  which  deliberate  continuity  of 
care  is  encouraged  and  reinforced. 

The  mental  health  statistician  is  in  an  excellent  posi- 
tion to  study  the  sufficiency  of  services  available  in  a 
community.  What  is  the  range  and  extent  of  mental 
health  services  available  in  the  community?  Where 
are  there  waiting  lists?  Are  there  facilities  which  are 
able  to  provide  a greater  amount  of  service  than  is 
demanded  of  them?  Do  certain  facilities  have  intake 
policies  which  seem  to  limit  unnecessarily  their  capa- 
city to  provide  service?  If  all  the  available  facilities 
in  a community  were  functioning  at  their  most  efficient 
rate,  what  would  be  the  capacity  of  the  entire  net- 
work to  absorb  new  cases?  Does  this  capacity  match 
the  rate  at  which  new  cases  appear  to  be  generated? 
If  not,  where  are  the  bottlenecks  and  how  can  they 
be  corrected?  For  what  kinds  of  patients  are  sufficient 
services  particularly  difficult  to  obtain?  If  new  serv- 
ices are  needed,  where  would  be  the  best  place  to 
locate  them  and  what  should  their  intake  policies  be? 
This  is  mental  health  program  planning  at  its  most 
fundamental  level,  and  the  statistician  can  assist  in 
gathering  the  facts  on  which  to  plan. 

Finally,  I want  to  talk  about  the  mental  health  statis- 
tician’s role  in  the  enhancement  of  quality  of  senice. 
I know  of  no  area  more  in  need  of  systematization  than 
the  measurement  of  treatment  and  program  success. 
As  budgets  become  larger  and  as  manpower  becomes 
scarcer  in  relation  to  the  population  at  risk,  the  need 
to  evaluate  effectiveness  increases  in  urgency.  The  issue 
is  simple.  What  is  your  mental  health  program  trying 
to  accomplish  and  how  successful  is  it  in  accomplish- 
ing it?  I do  not  need  to  remind  anyone  here  that  the 
Federal  Government,  in  collaboration  with  States  and 
local  communities,  has  embarked  on  a grand  experi- 


371-497  0—70 2 


3 


ment  in  the  reorganization  of  mental  health  services 
in  this  country.  This  grand  experiment  is  just  that.  It 
is  most  certainly  not  the  consequence  of  empirical 
studies  which  were  designed  to  evaluate  various  ap- 
proaches to  providing  psychiatric  care.  It  represents 
a philosophy  and  a strategy  and  both  the  philosophy 
and  the  strategy  are  thoroughly  unevaluated.  The 
philosophy  is  to  keep  patients  functioning  in  normal 
family  and  work  roles  insofar  as  these  are  relevant  and 
possible.  The  strategy  is  the  community  mental  health 
center — an  omnibus  agency  in  the  community.  Some 
of  the  consequences  of  this  strategy  are  fairly  clear — 
the  community  general  hospital  rather  than  the  State 
mental  hospital  will  become  the  focal  point  of  mental 
health  services,  many  State  mental  hospitals  will  be  con- 
verted into  community-oriented  comprehensive  service 
centers,  and  the  center  of  gravity  of  mental  health  prac- 
tice will  likely  move  somewhat  closer  to  the  medical 
physiological  end  of  the  physiological-psychosocial  con- 
tinuum. I am  enormously  enthusiastic  about  this  phi- 
losophy and  strategy  not  because  it  is  so  imaginative  or 
so  likely  to  be  successful,  but  because  it  may  lead  us 
closer  to  the  only  real  hope  in  the  field  of  psychiatry, 
namely  effective  primary  prevention.  As  presently  con- 
ceived, the  federal  community  mental  health  center 
movement  is  designed  essentially  to  improve  the  organi- 
zation of  the  delivery  of  a relatively  traditional  complex 
of  services.  We  have  heard  repeatedly  about  the  five 
anointed  services : inpatient  care,  outpatient  care, 
partial  hospitalization,  emergency  services,  and  consul- 
tation. I cannot  help  but  wonder  how  the  course  of 
American  psychiatric  history  might  have  been  changed 
had  the  five  required  services  of  a community  mental 
health  center  included,  for  example,  goal  and  time 
limited  individual  and  group  counseling,  anticipatory 
guidance  and  crisis  intervention,  social  system  analysis 
and  modification,  analytic  and  experimental  epidemi- 
ological studies,  and  mental  health  program  evaluation. 

There  is  some  research  coming  out  of  Los  Angeles 
County  Hospital  which  suggests  that  psychiatric  aides 
are  as  successful  as  psychiatrists  in  providing  meaning- 
ful help  to  hospitalized  patients,  that  treatment  of 
whatever  kind  is  not  demonstrably  better  than  no 
treatment  at  all  in  helping  psychiatric  patients  recover 
their  promorbid  level  of  adjustment,  and  that  virtually 
all  psychiatric  hospitalizations  are  unnecessary.  While 
our  professional  sights  are  clearly  set  toward  the  com- 
prehensive community  mental  health  center,  from  a 
research  and  evaluation  point  of  view  we  are  in  a period 
of  extraordinary  turmoil.  In  a state  of  programatic  tur- 
moil, one  solution  is  facts,  and  mental  health  stat- 


isticians must  help  in  gathering  and  interpreting  those 
facts. 

Just  what  are  the  facts  which  need  to  be  gathered 
in  order  to  evaluate  the  quality,  that  is,  the  effective- 
ness, of  our  mental  health  services?  The  facts  must 
concern  themselves  with  measuring  the  extent  to  which 
program  objectives  are  being  attained.  And  this  means 
that  there  have  to  be  some  explicit  objectives.  I have 
talked  to  this  point  on  other  occasions  (see  Bloom,  in 
press)  and  rather  than  pursue  this  road  today,  let  me 
suggest  that  there  also  needs  to  be  a systematic  evalua- 
tion of  individual  clinical  objectives,  that  is,  of  the 
treatment  success  with  respect  to  each  patient  who 
comes  in  contact  with  our  care-giving  system.  It  is  this 
kind  of  evaluation  I would  like  to  talk  about  today.  If 
you  walk  away  from  this  session  with  one  new  idea,  I 
hope  it  will  be  this.  At  the  end  of  every  diagnostic  eval- 
uation made  of  a patient  when  he  is  first  seen  in  a clini- 
cal facility  and  accepted  for  treatment,  two  questions 
must  be  always  .asked : ( 1 ) what  objectives  will  you 
have  in  treating  this  patient?  and  (2)  by  what  signs 
will  you  know  that  you  have  achieved  these  objectives? 
It  simply  won’t  do  to  evaluate  mental  health  programs 
by  some  global  service  statistics  such  as  the  extent  to 
which  hospitalizations  or  rehospitalizations  are  pre- 
vented or  shortened.  Hospitalization  is  often  the  best 
thing  that  can  happen  to  a patient  and  to  deny  him 
hospitalization  is  to  behave  in  a most  cavalier  fashion. 
Prevention  or  shortening  of  hospitalization  can  be  ac- 
complished simply  by  administrative  edict,  and  I have 
the  impression  that  this  is  often  exactly  what  is  done. 
The  real  question  is  what  happens  to  the  individual  pa- 
tient in  the  community  or  to  the  sum  of  the  psycho- 
pathology in  the  community  as  a consequence  of,  for 
example,  reducing  length  of  hospitalization.  We  must 
remember  that  troubled  people  have  been  denied  psy- 
chiatric care  for  years,  particularly  if  they  couldn’t  af- 
ford to  buy  it.  It  is  no  new  discovery  in  the  mental 
health  field  that  patients  survive  if  they  are  not  treated, 
or  treated  briefly,  or  treated  here  instead  of  there.  Sta- 
tistics on  admissions  and  discharges  and  on  length  of 
hospitalization,  whether  for  the  total  population  or  by 
age,  type  of  facility,  or  diagnosis  tell  us  virtually  nothing 
about  the  quality  or  effectiveness  of  our  services.  Even 
when  such  statistics  are  collected  over  time  so  that  we 
can  point  to  trends,  we  really  know  very  little  about  any 
changes  which  may  be  occurring  in  incidence  or  preva- 
lence, or  severity,  or  duration  of  the  major  emotional 
disorders  with  which  we  are  concerned. 

But  imagine,  if  you  will,  a community  mental  health 
program  in  which  these  two  questions  are  answered  for 
every  patient  at  the  start  of  his  treatment — what  are 
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you  trying  to  accomplish  with  this  patient,  and  how 
will  you  know  if  you  have  accomplished  it?  The  stage 
is  now  set  for  a major  and  definitive  study  of  treat- 
ment effectiveness  based  upon  unique  patients  each 
with  unique  treatment  goals.  I suspect  that  these 
studies  will  be  very  useful  at  the  national  level  even 
if  they  are  conducted  somewhat  differently  from  com- 
munity to  community,  even  if  there  are  no  standard- 
ized definitions  and  categories  of  treatment  objectives, 
or  standardized  lists  of  signs  of  treatment  success.  As  a 
matter  of  fact,  I would  urge  each  clinician  to  state  his 
objectives  in  his  own  words,  and  if  he  doesn’t  know 
why  he  is  seeing  a particular  patient,  let  him  say  so. 
Similarly,  let  him  state  the  evidence  whereby  he  will 
judge  the  success  of  his  clinical  work  in  his  own  way, 
and  again,  if  he  doesn’t  know  how  to  judge  it,  let  him 
say  so.  To  begin  with,  in  the  case  of  clinicians  who  do 
not  know  why  they  are  seeing  a patient,  or  who  do 
know  more  or  less  where  they  are  heading  but  don’t 
know  how  to  tell  when  they  get  there,  we  are  dealing 
with  clinical  activities  which  cannot  be  evaluated  and 
we  might  just  as  well  admit  it.  It  is  up  to  the  profes- 
sional staff  or  to  the  program  director  or  to  the  lay 
advisory  board,  or  the  county  commissioners  to  decide 
what  to  do  about  this  state  of  affairs.  They  could  decide 
to  tolerate  it,  to  encourage  it,  or  to  minimize  it.  But  it 
may  be  a sobering  realization  to  such  clinicians  to  know 
that  their  work  with  respect  to  some  particular  patient 
cannot  be  evaluated — they  have  undertaken  the  treat- 
ment of  a patient  with  entirely  unspecified  goals,  or 
with  goals  stated  so  abstractly  that  they  cannot  pin 
themselves  down  to  any  observable  points  of  reference 
in  order  to  determine  the  extent  to  which  the  goals 
have  been  achieved. 

If  objectives  and  signs  can  be  specified,  however, 
we  have  the  following  happy  state  of  affairs.  Every 
month  each  clinician  can  be  quizzed  with  respect  to 
each  of  his  active  patients.  “Are  these  reasons  which 
you  mentioned  a month  ago  for  seeing  this  patient  still 
applicable?”  “Are  these  signs  which  you  mentioned  a 
month  ago  as  the  ways  in  which  you  would  determine 
treatment  success  still  applicable?  If  not,  with  what 
objectives  in  mind  are  you  now  seeing  the  patient? 
What  indications  would  you  now  use  to  determine 
success?”  For  every  closed  case  each  clinician  could 
be  asked  to  what  extent  the  desired  behavior  changes 
actually  took  place  and  some  parallel  assessment  of 
degree  of  improvement  at  the  time  of  discharge.  Be- 
cause the  signs  of  improvement  are  observable,  they 
are  subject  to  verification.  Clinicians  could  also  be 
asked  about  behavioral  changes  they  might  anticipate 
appearing  in  the  future,  and  these  could  be  verified  at 


a later  date.  Now  here  are  some  of  the  types  of  analyses 
which  could  be  made  on  this  data  either  directly  or 
indirectly  related  to  quality  of  care : 

1.  What  is  the  general  nature  of  interclinician  reli- 
ability or  agreement  in  the  establishment  of  treat- 
ment goals? 

2.  In  the  case  of  patients  for  whom  there  are  particu- 
lar treatment  goals,  what  is  the  nature  and  range 
in  the  types  of  behavior  deemed  suitable  to  be  used 
to  index  goal  attainment? 

3.  Are  there  any  changes  over  time  in  the  treatment 
objectives  of  clinicians? 

4.  What  patient  or  therapist  factors  are  associated 
with  the  ability  to  establish  treatment  goals  and 
objective  signs  of  success? 

5.  Are  treatment  goals  related  to  diagnostic  charac- 
teristics of  patients  or  to  judgments  of  severity  or 
chronicity? 

6.  Are  certain  treatment  goals  easier  to  achieve  than 
other  goals? 

7.  In  what  proportion  of  cases  are  treatment  goals 
attained,  for  how  long  and  at  what  cost? 

8.  What  patient  characteristics  are  significantly  re- 
lated to  the  achievement  or  failure  in  the  achieve- 
ment of  treatment  goals? 

9.  What  are  the  consequences  to  the  goal  setting 
process  of  a feed-back  system  in  which  clinicians 
are  informed  regarding  the  general  degree  of  suc- 
cess of  their  various  treatment  goals? 

10.  Are  there  any  significant  differences  in  degree  or 
duration  of  goal  attainment  as  a function  of  various 
types  of  mental  health  treatment  programs?  Are 
certain  goals  more  easily  accomplished  by  State 
hospitals  than  by  community  mental  health  cen- 
ters? Are  other  goals  more  easily  accomplished  in 
the  community?  Are  certain  goals  achieved  with- 
out treatment  as  easily  as  with  treatment? 

In  my  judgment,  success  in  accomplishing  the  objec- 
tives of  a mental  health  program  must  be  measured 
primarily  in  terms  of  success  in  achieving  service  objec- 
tives with  each  client  in  the  system — whether  the  client 
is  a patient,  a relative  of  a patient,  or  a consultee.  There 
are,  of  course,  programmatic  objectives  in  addition  to 
clinical  objectives  and  perhaps  at  another  occasion  we 
can  talk  about  their  evaluation.  But  the  study  of  clinical 
objectives  and  their  vicissitudes  must  proceed  or  go  on 
concurrently  with  the  study  of  program  objectives. 

Well,  all  this  is  a tall  order  for  a mental  health 
statistician.  I have  described  a person  who  is  part 
statistician,  part  research  consultant,  part  demog- 
rapher, part  ecologist,  part  epidemiologist,  and  all 
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scientist.  This  formulation  may,  in  fact,  be  entirely 
unrealistic  and  perhaps  even  visionary.  But  dimly 
sensed  visions  of  the  future  can  help  us  in  planning 
programs  in  the  present.  Obviously  these  expanded 
roles  for  mental  health  statisticians  have  implications 
for  manpower  and  for  training.  We  need  more  and 
better  people.  We  need  specially  prepared  teaching 
materials,  perhaps  assembled  in  book  form.  Training 
programs  in  academic  settings  as  well  as  inservice  train- 
ing programs  need  to  be  generated  and  expanded. 
Academic  programs  such  as  the  one  shortly  to  be 
started  in  the  Department  of  Biostatistics  at  the  School 
of  Public  Health  at  the  University  of  North  Carolina 
under  the  direction  of  Bernard  Greenberg  need  to  be 
supported  and  encouraged.  Continuation  education 
programs  such  as  the  Western  Conference  on  Uses  of 
Mental  Health  Data  under  the  superb  leadership  of 
Charlie  Pettus,  or  the  more  senior  Southern  Regional 
Conference  on  Mental  Health  Statistics  under  Harry 
Williams’  able  and  dedicated  leadership  need  to  be 
extended  into  other  areas  of  the  country  as  well  as 


extended  in  scope.  Program  directors  of  hospitals  and 
of  community  mental  health  facilities  as  well  as  State 
mental  health  authorities  need  to  be  involved  in  the 
concerns  of  mental  health  statisticians  for  it  is  through 
them  that  your  staffs  will  be  expanded  and  upgraded. 
And  we  have  to  make  sure  that  our  work  is  not  ignored 
because  it  is  ignorable. 

There  is  no  doubt  that  we  are  entering  a new  era 
in  the  organization  of  mental  health  services  and  in  the 
integration  of  mental  health  with  general  health  pro- 
grams. If  this  new  era  is  not  to  suffer  the  same  ignomin- 
ious fate  as  the  great  hopes  of  the  past — bromides, 
lobotomy,  child  guidance  clinics,  hydrotherapy,  insulin, 
State  mental  hospitals,  perhaps  even  psychoanalysis,  we 
have  to  help  in  erecting  a solid  empirical  foundtation  to 
our  program  efforts.  We  need  data  thoughtfully  col- 
lected and  analyzed  and  intelligibly  presented,  respon- 
sive to  the  tasks  of  mental  health  program  planning  and 
program  evaluation.  I hope  that  all  of  you  will  be  able 
to  play  an  increasingly  important  role  in  this  endeavor. 
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DATA  SYSTEMS  AND  THEIR  APPLICATION  TO  CURRENT 
MENTAL  HEALTH  PROGRAMS 


Harold  L.  McPheeters,  M.D. 
Southern  Regional  Education  Board 


In  looking  at  some  of  the  current  efforts  and  current 
applications  of  mental  health  data  systems,  it  might 
be  well  to  trace  the  development  of  most  of  our  state 
mental  health  data  systems.  It  was  just  a shockingly 
short  time  ago,  it  seems  to  me,  no  more  than  20  years, 
that  there  were  no  substantial  requirements  any  place 
for  any  kind  of  broad  census  data  in  the  mental  hos- 
pitals and  institutions  for  the  retarded.  Census  data 
was  about  all  that  was  available  at  the  institutions  for 
the  mentally  retarded;  and  virtually  no  reporting  out 
of  clinics. 

The  only  substantial  use  made  of  anything  available, 
and  this  was  largely  just  crude  census  data,  was  to 
divide  the  census  data  figure  by  the  number  of  days 
in  a year  into  the  budget  figure  to  determine  the  insti- 
tution’s per  diem.  If  the  figure  was  low,  the  super- 
intendent could  brag  about  how  much  taxpayers’ 
money  he  had  saved.  I know  of  one  superintendent 
who  not  more  than  15  years  ago  publicly  bragged  that 
his  institution’s  per  diem  cost  had  never  exceeded  90 
cents  a day.  Imagine,  food,  clothing,  shelter,  and  treat- 
ment for  less  than  90  cents  per  day  per  patient.  And 
this  was  less  than  15  years  ago.  This  was  the  kind  of 
application  we  had  for  mental  health  data  systems 
20  years  ago. 

With  the  establishment  of  the  National  Institute 
of  Mental  Health,  a greatly  increased  stress  on  re- 
porting of  certain  data  regarding  movement  of  patients, 
age,  sex,  and  diagnoses,  came  into  being  in  mental 
hospitals,  schools  for  the  mentally  retarded,  and  mental 
health  clinics.  In  order  to  speed  the  reporting  of  com- 
parable data  from  State  to  State,  there  was  established 
the  Model  Reporting  Area  for  Mental  Hospital  Sta- 
tistics (MRA).  State  membership  in  the  MRA  re- 
quired the  use  of  common  definitions,  common 
reporting  periods,  and  common  reporting  forms.  Then, 
at  last,  it  was  possible  to  get  some  reasonably  trust- 
worthy comparisons  of  patient  movement,  diagnostic 
patterns,  and  costs  from  State  to  State. 


At  this  time  we  saw  the  publication  by  the  staff  of 
NIMH  of  several  articles  based  on  these  data.  These 
gave  us  a much  clearer  picture  for  the  first  time  of  the 
national  mental  health  picture,  but  I think  back  home, 
in  the  individual  States,  the  States  made  very  little  use 
of  this  data  within  their  own  systems.  However,  it  was 
at  this  time,  with  the  start  of  the  MRA  and  the 
national  level  of  interest  in  mental  health  data  that 
mental  health  statistics  units  came  to  be  established  in 
State  departments  of  mental  health. 

In  those  early  days,  the  main  purpose  in  having  a 
statistics  unit,  it  seemed  to  me,  was  to  gather  and 
prepare  the  data  for  the  Federal  reports.  No  one  quite 
knew  what  to  do  with  the  statistics  once  they  were 
compiled,  other  than  to  send  the  required  forms  to 
Washington. 

No  one  in  the  department  at  the  level  of  program 
administrators  or  clinicians  really  understood  the  mis- 
sion of  that  small,  windowless,  inside  office  that  was 
crowded  with  file  cabinets.  All  they  knew  was  that 
the  statistician  stayed  in  there  hovering  over  a clatter- 
ing IBM-101,  while  a glassy-eyed  clerk  sat  off  and 
gang-punched  zeros  into  stacks  of  IBM  cards,  but 
nobody  really  knew  what  this  was  all  about.  They 
did  things  over  there  and  sent  the  reports  in  to  Wash- 
ington, and  in  time  you  got  a report  back  out  of 
Washington,  a year  and  a half  later,  of  what  all  this 
data  was  about. 

Of  course,  you  could  still  divide  the  total  expendi- 
tures by  the  average  census  figure  and  come  up  with  a 
per  diem  figure  as  you  did  before  the  system  was  estab- 
lished but  by  this  time  an  early  form  of  creative  feder- 
alism was  already  doing  this  as  part  of  national 
reporting. 

One  new  and  useful  application  of  mental  health 
data  did  come  out  of  these  early  machine  systems.  The 
program  people  occasionally  called  on  the  statistician 
in  his  small,  windowless  office  to  do  some  kind  of  spe- 
cial runs  of  data  to  help  them  make  various  admin- 
istrative decisions  about  such  matters  as  whether  to 
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shift  certain  counties  in  the  hospitals’  catchment  area, 
to  another  hospital  district,  or  whether  the  alcoholic 
admissions  could  be  accommodated  in  ward  19  or 
would  have  to  take  over  ward  S because  there  were 
too  many  of  them,  and  ward  S was  a larger  ward, 
which  was  really  needed  by  the  hospital  for  old  ladies 
(females,  65  and  over) . In  this  way  the  new  statistical 
data  system  did  find  a new  use;  namely,  doing  special 
studies.  They  were  sporadic  but  important  studies 
used  in  programing  within  the  States.  In  most  cases, 
the  statistician  had  no  previous  inkling  that  a request 
for  data  was  to  be  made.  The  request  simply  showed 
up  in  a memo.  The  data  were  gathered  from  the  cards 
and  sent  off  with  no  further  word  of  it.  The  statisti- 
cian was  seen  only  as  the  deus-ex-machina  who  could 
twist  the  tails  of  all  those  IBM  contraptions  and  come 
up  with  appropriate  tables  and  figures  in  response  to 
the  memos. 

It  almost  never  occurred  to  anyone  in  the  Depart- 
ment that  the  statistician  might  be  initially  involved 
in  planning  and  analyzing  what  data  would  be  needed 
to  arrive  at  the  solution  to  the  problem.  Generally,  I 
think  the  statistician  was  perceived  to  be  like  the  bird 
in  the  cuckoo  clock  and  came  out  only  when  the  Fed- 
eral hand  passed  the  fiscal  12  and  announced  the  num- 
ber for  that  year’s  report. 

When  I got  to  New  York  a whole  new  concept  of 
what  a statistician  might  be  and  might  do  came  to  me 
in  seeing  the  way  the  statistician  operated  in  relation 
to  the  statistics  program  there.  In  New  York  I would 
not  have  considered  having  a meeting  of  something 
like  our  policy  committee,  which  met  weekly,  without 
having  the  statistician  present  where  he  could  feed  in 
some  of  the  insights  that  he  already  had  from  his 
knowledge  of  the  trends  and  patient  movement  and 
everything  else  in  the  system.  He  could  frequently  help 
us  formulate  the  questions  better,  and  frequently  had 
answers  already  available  from  his  knowledge  of  his 
system.  He  sent  a message  downstairs  and  could  come 
up  with  the  right  answer  right  off,  on  the  spot.  It  wasn’t 
a matter  of  waiting  until  the  meeting  was  over  and 
sending  a memo  down  asking  for  the  answer  to  the 
question.  The  idea  of  having  statisticians  in  as  part  of 
the  planning  mechanism  of  the  ongoing  operation  was 
something  that  had  really  not  occurred  to  me  before. 
My  statistician  in  Kentucky  sat  in  on  our  regular 
biweekly  staff  meetings  of  the  Department;  but  in  terms 
of  detailed  kinds  of  planned  activity,  I’m  afraid  it  was 
some  of  the  rest  of  us  in  the  Department,  administrative 
people,  and  program  people,  who  sat  down.  If  we 
thought  of  something  we  wanted  out  of  that  data  sys- 
tem over  there,  we  would  send  a memorandum  over 


and  get  that.  I think  this  question  of  application  of 
data  systems  in  most  places  depends  on  the  role  of  the 
statistician,  what  is  in  his  system,  and  what  capacity 
he  has  for  using  it. 

As  the  years  moved  on  our  IBM  machines  became 
larger,  faster,  and  more  versatile.  Now  we  have  the 
biggest  “gee-whiz”  of  them  all — the  computer.  It  is 
my  understanding  that  most  computers  in  use  today  in 
business,  industry,  and  in  Government  are  used  almost 
exclusively  to  handle  one  of  two  kinds  of  data  prob- 
lems, one  to  provide  speed  and  the  other  to  process  a 
large  quantity  of  data.  Perhaps  the  major  use  made  of 
them  by  mental  health  departments  is  to  provide  for 
handling  of  the  large  quantities  of  data  that  are  accu- 
mulated on  our  thousands  of  patients. 

More  recently,  program  administrators  and  clinicians 
have  come  to  realize  that  the  same  ability  of  the  com- 
puters to  handle  large  amounts  of  patient  data  may  be 
used  in  other  data  systems,  such  as  personnel,  payroll, 
fiscal,  inventory,  and  facility  data.  In  most  States  each 
one  of  these  data  systems  is  set  up  as  a separate  system 
and  tends  to  remain  very  separate  when  the  various 
systems  are  put  on  computers.  In  only  a very  few 
places  do  we  hear  talk  of  establishing  linkages  between 
the  various  data  systems.  Essentially  we  are  doing  very 
little  more  than  when  we  were  in  the  punch-card 
machine  days,  and  yet  the  computer,  in  addition  to  its 
ability  to  rapidly  handle  large  amounts  of  data,  also 
has  the  ability  to  show  relationships.  This  requires 
much  more  sophisticated  planning  and  programing 
than  most  of  us  are  now  prepared  to  give,  but  it  remains 
as  a tremendous  possibility  which  we  should  explore 
and  develop.  For  example,  in  New  York,  before  the 
computer  was  to  be  used  a lot  of  planning  was  done 
in  this  area,  and  one  of  the  applications,  for  instance — 
I’m  sure  dozens  more  have  been  added — was  a mecha- 
nism which,  when  a patient  came  in  on  an  order  for 
some  sort  of  action  to  be  taken  within  a certain  period 
of  time,  fed  this  information  into  the  computer  and  the 
computer  listed  it  for  outgo  on  a list  of  patients  for 
whom  some  action  had  to  be  taken  10  days  hence  or 
something,  so  this  could  be  fed  back  to  the  hospital. 
Here  was  an  example  of  the  computer  being  used  in  a 
way  to  do  something  for  you  other  than  just  tabulations 
and  a question  of  speed,  but  this  sort  of  thing  is  really 
extremely  rare. 

What  is  the  role  then  of  the  current  mental  health 
patient-data  statistician?  At  present  in  most  places  it 
seems  to  me  his  role  remains  pretty  much  unchanged 
from  machine  days.  He  is  still  too  often  the  bird  in  the 
cuckoo  clock  who  makes  up  the  Federal  report,  and 
who  responds  to  memos  from  the  administration.  He 
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seldom  has  any  firm  notion  about  what  goes  on  in  those 
other  data  systems  of  his  program — personnel  data, 
fiscal  data,  inventory  data,  etc.  He  often  does  not  see 
himself,  nor  is  he  often  seen  by  others,  as  having  any 
broader  role  to  play  in  the  Department’s  program  than 
the  one  he  now  plays. 

What  are  some  of  the  consequences  of  this?  The 
most  unfortunate  consequence  is  that  the  data  now 
available  is  not  being  used  to  anything  like  its  potential 
in  developing  our  programs  of  patient  care,  prevention, 
and  rehabilitation.  In  addition,  our  data  systems  tend 
to  be  static  rather  than  constantly  evolving  to  greater 
levels  of  usefulness.  Finally,  our  data  systems  remain 
essentially  unrelated  to  existing  program  operations. 
Program  people  are  not  really  using  our  data  systems 
for  monitoring  operations,  for  development,  for  evalu- 
ation, etc.,  as  they  really  might  be  used. 

But  just  what  should  be  the  applications  of  data 
systems  to  mental  health  program  operations?  A cer- 
tain research  or  evaluation  function  is  implicit  in  any 
program’s  operation.  A systematic  collection  of  data, 
analysis  and  presentation,  having  as  their  main  pur- 
pose helping  administrators  to  obtain  and  spend 
money  wisely  and  effectively  in  managing  the  prob- 
lems of  prevention,  treatment  and  rehabilitation  of 
emotional  disturbance,  and  mental  retardation.  In  an 
operating  agency  this  data  collection,  analysis  and 
presentation  function  serves  essentially  five  purposes — 

First — surveillance, 

Second — recordkeeping, 

Third — evaluation, 

Fourth — development, 

Fifth — basic  research. 

To  some  extent,  as  we  move  from  surveillance  to  basic 
research,  we  go  through  increasingly  complex  and  so- 
phisticated data  systems,  but  this  is  exactly  what  our 
data  systems  should  be  doing — steadily  evolving  into 
more  sophisticated  and  more  useful  systems.  It  is  neces- 
sary to  start  at  simple  levels,  but  let  us  not  be  content  to 
stay  at  those  levels.  Let’s  look  a little  more  specifically 
at  these  items.  Surveillance  involves  determining  the 
size,  nature,  and  extent  of  the  problems  we  are  at- 
tempting to  manage.  As  with  all  data  systems,  we  can- 
not hope  to  cover  every  aspect  of  the  problem.  We  must 
be  selective  of  what  items  we  keep  accounts  on,  but 
these  systems  should  be  steadily  improved,  increased, 
and  developed  from  these  simple  things.  We  need  to 
look  more  at  community  needs  and  activities.  Most  of 
our  current  patient  data  systems  in  the  mental  hospitals, 
schools  for  the  retarded,  and  clinics  are  concerned  only 
with  those  who  are  admitted  to  the  facility,  making  no 


attempt  to  tell  us  anything  about  the  extent  of  prob- 
lems in  the  community. 

Perhaps  if  we  developed  reporting  systems  for  cases 
about  which  we  consult  but  do  not  actually  see  or 
admit  to  our  services  we  could  formulate  some  notions 
about  the  dimensions  of  the  problems  in  the  community 
and  which  portions  of  the  problems  we  are  actually 
serving  in  our  programs.  Furthermore,  consider  that 
our  categories  of  patients  and  the  movement  are  pres- 
ently quite  crude;  the  diagnostic  categories  have  been 
flayed  many  times  before  because  of  their  vague  and 
varying  use,  but  still  we  use  them.  There  has  been  some 
work  done  with  categories  of  presenting  problems  or 
symptoms  rather  than,  for  instance,  diagnostic  cate- 
gories, but  this  still  remains  largely  an  experimental 
kind  of  activity.  It  seems  to  me  this  kind  of  thing  will 
provide  a step  in  the  direction  of  increased  specificity 
and  sophistication.  I think  the  kind  of  program 
WICHE  (Western  Interstate  Commission  for  Higher 
Education)  has  with  their  mental  retardation  data  sys- 
tem furnishes  them  with  a great  deal  more  information 
about  the  pressing  problems  of  kids  in  their  13-States’ 
institutions,  such  as  the  kinds  and  extent  of  the  prob- 
lems, the  numbers  that  are  bedfast,  ambulatory,  toilet 
trained,  etc.  This  computerized  reporting  is  a step  in 
the  direction  of  increased  sophistication;  and  at  times 
maybe  they  have  felt  they  made  the  system  so  sophis- 
ticated it  overwhelmed  them  with  bits  of  information 
they  did  not  quite  know  what  to  do  with,  but  never- 
theless the  kind  of  stuff  essential  to  know,  in  a more 
sophisticated  way,  about  the  extent  of  the  problems 
with  which  we  are  dealing.  For  instance  we  all  know 
there  have  been  increases  in  the  numbers  of  children 
and  adolescents  coming  into  our  hospitals.  The  sur- 
veillance has  made  us  aware  of  this.  We  know  there 
are  more  alcoholics  coming  in,  and  that  syphilitics  have 
almost  disappeared.  Our  program  should  be  retailored 
in  terms  of  these  different  kinds  of  problems  that  we 
are  trying  to  deal  with,  if  the  system  is  really  doing  its 
job  in  surveillance. 

The  second  objective  is  recordkeeping.  This  involves 
data  about  the  program’s  operation  and  the  outcome 
of  its  efforts.  Here,  too,  some  decisions  about  what  data 
to  keep  must  be  made,  depending  on  program  and  ad- 
ministrative needs  and  outside  demands  such  as  budget 
officers  or  legislatures,  etc.  The  recordkeeping  func- 
tions should  also  be  constantly  improved  and  devel- 
oped. For  instance,  presently  almost  no  data  is  kept  on 
treatments  given  or  rehabilitation  measures  taken. 
Treatment  data  simply  is  not  in  our  system.  All  we 
know  really  is  length  of  stay  and  some  vague  com- 
ment about  improved  or  unimproved,  which  really 
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does  not  tell  us  much  about  outcome.  These  are  prob- 
ably the  levels  at  which  we  n'eed  to  begin,  but  isn’t 
it  time  to  move  to  greater  sophistication  at  this  point? 
Particularly  the  treatment  data  should  be  added  to 
our  data  systems,  and  outcome  should  preferably  be  in 
terms  of  the  problems  presented  (that  is,  what  have 
we  actually  done  in  terms  of  the  problem  presented) 
rather  than  in  terms  such  as  improved  or  unimproved. 

I think  it  significant  that  most  of  our  data  today  is 
called  “patient  movement  data,”  for  that  is  about 
all  that  it  is.  It  certainly  is  not  treatment  data  or  out- 
come data  or  any  kind  of  clinical  data,  but  we  should 
have  this  kind  of  information  if  data  systems  are  going 
to  be  maximally  useful.  Very  few  of  our  State  mental 
health  data  systems  have  currently  evolved  much  be- 
yond the  rather  rudimentary  stages  of  these  first  two 
functions — surveillance  and  recordkeeping.  These  are 
essentially  what  our  systems  are  doing  at  this  point. 

The  next  data  function  is  evaluation  which  should 
be  a major  application  of  any  data  system.  This  func- 
tion requires  the  setting  of  some  specifications,  what 
we  intend  to  do,  and  a certain  rigor  in  the  gathering  of 
data.  Too  few  program  administrators,  psychiatrists, 
and  other  clinicians,  are  aware  of  the  possible  uses  of 
data  systems  for  evaluation.  It  appears  that  maybe  the 
same  thing  is  true  with  statisticians. 

Obviously,  if  data  systems  are  to  be  used  for  evalu- 
ation, program  people  and  statisticians  must  work  to- 
gether to  set  the  specifications  and  to  apply  the  data- 
gathering  approaches.  Who  should  take  the  lead  in 
this?  Sometimes  statisticians  argue  that  this  is  not  a 
proper  role  for  a statistician,  and  in  a technical  sense 
they  may  be  right — a statistician’s  training  is  formally 
only  in  statistical  procedures,  not  in  their  use.  The  same 
argument,  of  course,  might  equally  hold  true  for  psy- 
chiatrists, psychologists,  and  others  who  administer  the 
programs.  They  are  trained  in  clinical  psychiatry,  clin- 
ical psychology,  and  not  trained  in  program  evaluation 
nor  administration.  But  somebody  has  to  take  the  lead 
in  this.  I don’t  know  who  it  should  be,  but  would  like 
to  see  it  more  often  be  the  statistician.  Evaluation  also 
evolves  a degree  of  sophistication. 

The  significant  steps  in  evaluation  are,  first  of  all, 
setting  the  objectives  of  the  program  in  some  fashion 
that  can  be  measured.  At  first  these  objectives  are  going 
to  have  to  be  arrived  at  by  some  kind  of  concensus 
which  is  refined  to  more  ultimate  levels.  When  you 
sit  down  with  your  clinician  and  talk  about  what  our 
goals  are  going  to  be  for  alcoholics — is  it  going  to  be 
sobriety,  drying  them  out,  some  sort  of  treatment,  or 
some  kind  of  control  of  the  drinking  but  not  absolute 
abstention?  You  have  to  have  some  kind  of  goal  and 


some  notion  of  how  many  people  you  plan  to  treat  in 
the  course  of  a certain  period  of  time.  These  kinds  of 
objectives  need  to  be  set.  As  a second  step  the  input 
and  output  variables  have  to  be  determined.  Initially 
these  will  probably  be  very  simple  input  variables  of 
staff  allocations  of  services,  number  of  psychiatrists  on 
the  service,  number  of  psychologists,  etc.,  but  later  they 
will  be  evolved  into  more  sophisticated  time  and  effort 
variables. 

Output  variables  will  also  evolve  from  very  primi- 
tive concepts,  such  as  “released”  to  something  such  as 
“successful  employment”  or  “independent  social  liv- 
ing.” We  will  also  have  to  have  other  more  sophisti- 
cated measures  of  successful  outcome.  A concensus  on 
the  ways  to  measure  these  variables  has  to  be  reached. 
What  kind  of  criteria  are  we  going  to  accept  as  the 
standard  for  successful  employment,  for  instance,  or 
independent  social  living?  These  criteria  will  initially 
be  judgmental  and  crude,  but  later  they  should  evolve 
into  more  systematic  and  formal  measures  as  time 
goes  along. 

Moving  on  to  development,  we  use  the  data  sys- 
tems for  planning  of  projects  and  programs  largely 
that  do  not  now  exist  or  that  are  being  substantially 
modified.  The  steps  we  use  are  a combination  of 
those  listed  under  surveillance  and  recordkeeping  and 
evaluation,  with  the  exception  that  here  is  an  oppor- 
tunity to  practice  intervention  research,  to  institute  a 
new  program,  and  to  measure  its  step  by  step  progress 
to  productivity  and  efficiency.  It  comes  closer  to  the 
experimental  research  model,  in  that  you  can  have 
some  controls.  You  can  start  this  in  certain  hospital 
districts,  schools,  or  programs  and  not  in  others,  so  you 
have  some  measures  of  what  would  happen  if  you  did 
nothing  compared  to  what  you  are  doing  in  the  sys- 
tem in  which  you  are  intervening.  But  the  data  system 
has  to  be  set  up  to  report  its  data  back  to  the  adminis- 
trator at  very  frequent  intervals  so  that  he  knows 
whether  he  is  coming  close  to  his  goal  or  not,  and 
what  in  the  system  needs  to  be  changed  or  modified 
in  order  to  help  him  reach  the  goal. 

In  basic  research,  the  researcher  tries  to  use  the  data 
system  to  see  beyond  the  immediate  problems  to  more 
fundamental  problems.  The  research  is  based  on  rath- 
er carefully  formulated  theory,  not  just  on  trial  and 
error  observations.  Most  operating  agencies  do  not 
support  this  kind  of  basic  research  in  their  regular 
operations,  but  it  is  the  kind  of  use  of  the  data  sys- 
tem which  may  ultimately  have  the  greatest  payoff. 
This  level  of  basic  research  obviously  requires  an  in- 
terested and  motivated  clinician  who  can  develop  the 
theory  or  hypothesis  and  the  method  for  testing  it.  The 


10 


statistician  should  be  involved  in  the  planning  and 
analysis  of  the  data  as  well  as  in  its  collection. 

In  basic  research,  as  in  other  functions  of  data  sys- 
tems, we  must  expect  to  make  a beginning  at  a fairly 
primitive  level  and  gradually  evolve  to  more  sophisti- 
cated development  of  such  data.  This  basic  research 
is  usually  performed  in  a university  or  clinical  setting 
where  the  experimental  model  can  be  developed  with 
appropriate  controls  and  control  of  variables.  There 
is  usually  no  analysis  or  reporting  of  data  to  anyone 
until  the  experimental  period  is  completed.  Then  the 
data  is  tabulated  and  analyzed,  and  the  codes  revealed 
to  complete  the  experiment.  The  report  is  written  and 
sent  to  the  administrator  for  publication.  It  is  only 
when  this  final  report  is  revealed  that  its  failure  or 
success  is  known  and  why  it  failed  or  succeeded.  Un- 
like the  development  model  where  the  data  is  con- 
stantly reported  back  for  comparison  with  specified 
goals  so  that  program  needs  for  modification  can  be 
made  as  you  go  along;  this  experimental  approach  has 
limited  value  for  the  administrator  as  a regular  re- 
porting mechanism,  since  the  data  is  revealed  only  after 
the  experiment.  It  is  very  valuable  in  answering  certain 
basic  questions  about  treatment  procedures  and  the 
like,  but  data  systems  should  be  prepared  to  participate 
in  experimental  research  of  this  kind. 

In  all  of  these  functions  of  data  systems,  we  must 
remember  that  our  patient  systems  are  essentially  con- 
cerned with  what  most  systems  analysts  would  call  the 
output  portion  of  the  program;  that  is,  the  services 
we  are  rendering  to  people  and  the  kinds  of  adjust- 
ments in  outcomes  the  people  have.  When  we  focus 
exclusively  on  this  part  of  the  program,  we  may  be 
able  to  tell  something  of  the  outcome  or  effectiveness 


of  our  program  in  terms  of  the  people  served,  especially 
as  our  measures  become  more  sophisticated;  but  until 
we  include  the  input  data- — personnel,  budgets,  equip- 
ment, and  facilities — we  will  never  be  able  to  decide 
the  efficiency  of  the  whole  mental  health  program.  We 
may  be  able  with  the  sophisticated  criteria  to  measure 
effectiveness  but  not  efficiency  unless  we  use  the  input 
measures.  With  our  computers  it  becomes  operationally 
feasible  to  tie  the  input  and  output  data  systems  to- 
gether. It  will  require  a great  deal  of  thought  and 
planning,  between  program  people,  administrators, 
and  statisticians,  but  it  can  be  done  and  it  must  be 
done  if  our  mental  health  administrators  are  to  know 
whether  they  are  using  their  money  wisely  and  effec- 
tively. We  can  say,  yes,  we  are  treating  effectively  20 
patients,  but  maybe  these  20  patients  are  being  treated 
at  too  high  a cost.  The  treatment  may  be  tremendously 
inefficient  and  we  need  to  find  out  why. 

What  concerns  me  most  in  this  is  the  need  for  a 
person  in  the  program  who  thinks  and  functions  in 
terms  of  serious  applications  of  the  data  in  the  more 
advanced  functions  of  evaluation,  development,  and 
research,  as  well  as  in  surveillance  and  recordkeeping 
functions.  Shall  this  person  be  the  statistician  who  will 
be  a significant  leader  in  the  program’s  administration, 
in  planning  and  development,  or  will  the  statistician 
continue  to  be  the  bird  in  the  cuckoo  clock  wrho  fills 
out  the  federal  report  and  answers  occasional  memos. 
Time  will  tell,  but  I believe  it  is  time  for  us  to  stop 
merely  counting  our  programs  and  find  out  instead 
whether  our  programs  really  count.  Statisticians  should 
move  from  being  merely  technicians  to  becoming  lead- 
ers in  the  applications  of  their  data  systems  to  the 
agencies’  programs  and  problems. 


371-497  0—70 3 


11 


THE  NEW  HAVEN  CENSUS  USE  STUDY— A SMALL-AREA-DATA 

RESEARCH  PROJECT 

Caby  C.  Smith 
Bureau  of  the  Census 


The  first  U.S.  census  was  taken  in  1790.  Since  then 
there  have  been  17  censuses,  and  except  for  the  last 
one  in  1960,  they  were  all  done  in  the  same  way — by 
personal  visit  to  each  household. 

In  1960  the  U.S.  Post  Office  delivered  the  census 
questionnaires  in  advance  of  census  day.  Householders 
were  asked  to  complete  the  questionnaires  and  to  hold 
them  until  census  interviewers  visited  to  collect  the 
information. 

In  1970,  for  the  first  time,  the  Census  Bureau  will 
use  a mail-out/mail-back  data  collection  procedure  in 
the  143  largest  SMSA’s  and  other  urbanized  portions 
in  the  country  where  over  60  percent  of  our  population 
lives.  The  households  in  the  remainder  of  the  country 
will  also  receive  their  questionnaires  by  mail,  but  will 
be  asked  to  complete  it  and  hold  it  for  the  enumerator 
to  pick  up.  There  are  three  byproducts  of  the  1970 
mail  census  which  directly  or  indirectly  benefit  small- 
area-data  users.  These  are : 

1.  Metropolitan  Maps 

The  Census  Bureau’s  Geography  Division  is  preparing 
standard  scale  “Metro”  maps  for  all  of  the  SMSA’s  in 
the  country. 

2.  Address  Registers 

Mailing  addresses  for  the  mail  SMSA’s  will  be  acquired 
by  the  Census  Bureau  and  these  addresses  will  be  up- 
dated by  the  U.S.  Post  Office  just  prior  to  mailing  the 
1970  Decennial  Census  questionnaires.  Such  address 
lists  are  called  “Address  Registers.” 

3.  Address  Coding  Guides 

Obtaining  census  information  by  mail  instead  of  mak- 
ing personal  visits  has  made  it  necessary  for  the  Census 
Bureau  to  develop  a method  of  relating  data  for  an 
address  to  the  correct  census  block  and  tract  for  tabu- 
lation purposes — an  address  coding  guide.  A coopera- 
tive arrangement  between  the  Housing  and  Urban 


Development  Department  and  the  Census  Bureau  was 
developed  to  create  these  address  coding  guides.  Local 
planners,  with  HUD  financing,  are  using  census  work- 
sheets with  address  ranges,  census  metro  maps,  and  with 
local  address  source  material  are  creating  records  which 
assign  address  ranges  to  census  block-faces  or 
block-sides. 

Both  the  metro  maps  and  the  address  coding  guides 
will  be  made  available  to  anyone  in  the  mail  census 
SMSA’s.  The  address  register  is  confidential  and  will 
not  be  released.  It  may  be  used,  however,  by  the  Census 
Bureau  to  meet  certain  special  requests,  such  as  address- 
to-address  computer  matching,  or  as  a source  of 
addresses  for  sample  surveys. 

The  availability  of  these  census  byproducts  combined 
with  the  growing  need  for  small  area  data  and  im- 
proved accessibility  to  census  data  led  to  the  creation 
of  a Census  Advisory  Committee  on  Small-Area-Data. 
One  of  the  first  acts  of  this  committee  was  to  recom- 
mend establishing  a census  research  study  in  a city 
where  census  pretest  was  conducted  in  April  1967  in 
the  New  Haven,  Conn.,  SMS  A.  With  financial  support 
and  use  of  the  city’s  IBM-360  Model  30  from  Mayor 
Lee  of  New  Haven,  and  the  Federal  agencies  repre- 
sented on  the  advisory  committee,  the  New  Haven 
census  use  study  office  opened  in  September  1966.  Cur- 
rent financing  will  allow  the  use  study  to  continue  into 
fiscal  year  1970,  and  there  are  good  indications  that  it 
may  remain  in  New  Haven  to  work  with  1970  data. 
General  financial  participation  and  special  project 
funds  amount  to  well  over  a million  dollars.  General 
financial  participation  from  Federal  departments  comes 
from: 

— The  Department  of  Transportation, 

— The  Department  of  Health,  Education,  and 
Welfare, 

— -The  Department  of  Housing  and  Urban  Devel- 
ment, 

— The  Department  of  Commerce, 
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— The  Department  of  Defense. 

Agencies  supporting  special  projects  are : 

— The  Bureau  of  Public  Roads, 

— The  Children’s  Bureau, 

— The  National  Institute  of  Mental  Health, 

— The  Office  of  Civil  Defense. 

We  expect  that  other  Federal  departments  will  want 
to  participate  in  the  census  use  study  in  the  near  future. 

The  New  Haven  census  use  staff  is  composed  of 
15  professional,  technical  and  clerical  personnel,  with 
four  additional  staff  members  whose  salaries  are  paid 
by  local  and  Federal  agencies.  Two  Yale  University 
students  are  working  as  programers  and  two  consult- 
ants are  assisting  us:  Mr.  Edward  Hearle  of  Booz, 
Allen  and  Hamilton;  and  Mr.  John  Sweeney  of  Cam- 
bridge Computer  Associates.  We  truly  have  a Federal- 
State-local  cooperative  effort  in  New  Haven. 

Immediately  after  the  use  study  office  opened,  the 
staff  began  meeting  with  Federal,  State,  city,  university, 
and  private  groups  to  acquaint  them  with  the  nature 
of  a census  and  the  objectives  of  the  use  study.  Over 
200  meetings  have  been  held  in  an  attempt  to  get 
well-defined  needs  for  census  products  and  clear  speci- 
fications for  programing  special  tabulations  from  the 
basic  data  record  tapes.  Special  projects  or  requests 
for  tabulations  were  only  accepted  if  they  were  sig- 
nificant in  terms  of  the  study’s  objectives  and  if  the 
product  would  be  generally  applicable  in  cities  across 
the  Nation. 

The  objectives  of  the  census  use  study  are : 

1.  Determine  both  the  current  uses  and  likely  future 
needs  for  census  data  in  existing  local,  State,  and 
Federal  programs. 

2.  Develop  a system  that  will  allow  efficient  merging 
of  census  data  with  other  local  and  State  data  to 
meet  the  needs  of  specifically  defined  programs  with 
the  confidentiality  of  census  information  fully 
protected. 

3.  Investigate  the  benefits  of  cooperative  data  collec- 
tion between  census  and  other  local,  State  and 
Federal  groups  with  special  emphasis  on  personnel 
requirements,  preparation  of  guides,  and  develop- 
ment of  uniform  terms  and  procedures. 

4.  Investigate  the  level  of  detail  and  form  in  which 
census  data  should  be  made  available  to  local  users. 
This  will  include  investigating  ways  in  which  the 
Census  Bureau’s  tabulation  program  for  small  area 
statistics  can  be  improved  and  how  special  tabula- 
tions and  summary  data  on  magnetic  tape  can  be 
made  readily  available,  at  low  cost,  to  meet  local 
needs. 


5.  Develop  a package  of  computer  and  statistical  pro- 
grams for  use  by  local  committees  to  allow  rapid 
conversion  of  1970  Decennial  Census  data  into 
forms  more  useful  for  local  analysis. 

6.  Test  and  implement  various  applications  of  pro- 
posed census  mapping  and  computer  graphics  tech- 
niques using  local  and  census  data. 

7.  Analyze  the  results  of  the  study  for  potential  strate- 
gies to  be  incorporated  in  the  1970  census  plans,  and 
in  local  community  programs  to  take  advantage  of 
census  and  local  data. 

These  objectives  fall  into  four  major  avenues  of 
research : 

1.  Tabulations, 

2.  Matching, 

3.  Mapping  and  computer  graphics, 

4.  Feasibility  surveys. 

In  addition  there  are  special  projects  which  may  uti- 
lize some  or  all  of  these  research  categories. 

Tabulations 

Dozens  of  requests  for  tabulations  of  the  New  Haven 
census  pretest  “complete  count”  and  sample  data  have 
been  received  during  the  past  several  months  from 
city  and  State  agencies  and  private  groups.  Requests 
received  to  date  include : ( 1 ) different  age  and  income 
breakdowns  for  the  poverty  programs  and  educational 
interests;  (2)  persons  per  room,  by  age  and  race,  in 
occupied  units,  as  a measure  of  overcrowding,  and  (3) 
special  housing  tabulations,  by  income  and  rent,  for 
the  elderly.  As  we  receive  requests  for  tabulations  they 
are  processed  according  to  type,  area  unit,  and  whether 
it  is  feasible  to  produce  the  data  from  a programing 
and  cost  standpoint,  that  is,  the  “everything  by  every- 
thing” requests  are  not  processed  unless  more  meaning- 
ful specifications  can  be  developed.  Requests  for  tabu- 
lations are  coordinated  and  combined  when  possible 
with  those  of  other  groups. 

The  census  basic  record  computer  tapes  have  been 
converted  to  IBM  format  at  the  Bureau  for  use  on 
New  Haven’s  IBM-360,  Model  30,  or  IBM-7094’s 
at  Harvard  or  Yale  Universities.  These  tapes  will  be 
used  for  producing  special  tabulations  according  to 
locally  defined  needs. 

The  data-text  system,  developed  by  the  Department 
of  Social  Relations  at  Harvard,  will  be  used  to  produce 
some  of  the  tabulations  from  the  Census  Bureau  s basic 
record  tapes.  Relatively  inexperienced  programers  can 
be  used  with  this  system  to  quickly  produce  data  from 
a variety  of  requests. 
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One  of  the  primary  problems  of  producing  tabula- 
tions is  to  assure  that  information  about  the  individual 
or  household  is  not  disclosed.  The  Census  Bureau  has 
established  a Confidentiality  Committee  to  study  the 
various  disclosure  problems  involved  in  releasing  small 
area  data. 

Using  the  Bureau’s  address  coding  guide,  individual 
addresses  can  be  identified  to  block-faces  and  these 
block-faces  can  be  used  as  “building  blocks”  to  create 
local  administrative  study  and  project  areas  for  which 
census  data  is  needed  for  planning  and  evaluation  pur- 
poses. Previously  it  was  only  feasible  to  produce  de- 
tailed tabulations  for  standard  census  tracts  or  large 
political  units.  New  Haven  data  users  are  submitting 
area  dictionaries  describing  their  own  project  or  study 
areas  for  which  detailed  data  are  needed.  Some  of  the 
area  dictionaries  are: 

1.  Traffic  zones  (approximately  one-sixth  tract  size). 

2.  Community  action  program  and  redevelopment 

project  areas. 

3.  School  attendance  areas. 

4.  Telephone  cable  or  market  areas. 

Tabulations  by  small,  locally  defined  areas  will  be 
valuable  in  evaluating  current  programs  and  in  defin- 
ing new  project  areas. 

Matching  Studies 

One  of  the  basic  objectives  of  the  use  study  is  to 
develop  the  computer  programs  for  effectively  relating 
local  data  to  census  data  through  geographic  coding 
of  street  addresses  using  the  address  coding  guide  and 
address  register.  The  techniques,  when  developed, 
would  also  allow  the  merging  of  two  local  files  of  data. 
A two-stage  matching  program  is  now  fully  operational 
in  New  Haven.  The  first  stage  preprocesses  the  street 
address  into  a fixed  format  in  preparation  for  the 
matching  operation. 

Several  matching  studies  have  been  done  and  others 
are  planned.  They  fall  into  three  separate  categories: 
area,  address,  and  individual.  The  area  match  involves 
geographically  coding  to  block  face  the  addresses  from 
local  records  using  the  address  coring  guide.  Matching 
at  the  address  level,  e.g.,  vital  records  addresses  to 
census  data,  require  the  use  of  the  address  register. 
Matching  to  the  individual  level,  e.g.,  relating  census 
data  to  individual  hospital  or  clinic  records,  requires 
the  basic  characteristics  of  the  indvduals  from  both 
records.  These  characteristics,  sex,  age,  race,  relation- 
ship, etc.,  are  used  for  matching  since  the  census  basic 
tapes  do  not  contain  unique  identification  of  the 
individual. 


Some  of  the  area  and  address  matches  done  to  date 
are: 

■ — State  birth  records, 

- — Housing  complaints, 

— Geographic  coding  of  residence  of  New  Haven 
high  school  students, 

• — Police  complaints, 

— Annual  school  census  records. 

After  the  matching  operation,  the  census  data  will 
always  be  aggregated  to  a level  where  census  confiden- 
tiality rules  are  not  violated. 

Mapping  and  Computer  Graphics 

Another  basic  objective  of  the  census  use  study  is 
to  experiment  with  new  and  different  ways  of  dis- 
playing census  and  local  data  using  computer  graphic 
techniques.  In  order  to  do  mapping  experiments,  a 
geographic  base  file  had  to  be  developed.  Donald 
Cooke  and  William  Maxfield  of  the  census  use  study 
staff,  with  advice  from  the  Bureau’s  Statistical  Re- 
search Division,  created  the  DIME  (Dual  Independ- 
ent Map  Encoding)  file  to  serve  this  need. 

To  prepare  a DIME  file,  census  metro  maps  and 
local  address  sources  are  needed.  All  map  features  are 
divided  into  three  groups  ( 1 ) lines,  e.g.,  streets,  rail- 
road tracks,  city  limits,  streams;  (2)  land  areas,  iden- 
tified by  census  tract-block  numbers,  and  (3)  nodes, 
e.g.,  points  where  lines  intersect,  the  ends  of  deadend 
streets,  or  points  of  significant  curvature  of  lines.  The 
nodes  are  numbered  and  transcribed  onto  coding 
sheets  along  with  street  names,  census  tract  and  block 
numbers,  and  street  address  ranges  between  nodes. 
The  information  is  then  keypunched  and  in  a separate 
operation  coordinates  are  assigned  to  the  node  num- 
bers with  automatic  equipment.  The  combined  infor- 
mation is  then  transferred  to  magnetic  tape  where 
computer  edits  are  run  to  detect  errors,  such  as  un- 
bounded blocks,  so  that  a predictable  level  of  accur- 
acy can  be  achieved.  A reproduction  of  the  map  can 
then  be  made  by  computer-driven  plotting  equipment. 

We  are  currently  producing  computer  programs  for 
assembling  “tailored”  statistics  and  maps  for  locally 
defined  special  areas.  Computer  mapping  techniques 
using  the  pen  plotter,  printer  (SYMAP  and  MAP  01), 
the  geospace  plotter,  and  cathode  ray  tube  with  cen- 
sus and  local  data  are  being  tested. 

The  DIME  system  is  now  being  used  in  New  Haven 
and  DIME  files  have  recently  been  prepared  for 
Roanoke,  Va.,  and  Binghamton,  N.Y.,  to  further  test 
the  procedures  and  programs  so  that  the  DIME  sys- 
tem can  be  made  into  a do-it-yourself  package  for 
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planners  who  wish  to  do  address  matching,  data  dis- 
play, and  tabulations  of  local  or  census  data. 

The  final  system  will  be  made  available  to  planners 
across  the  Nation  who  need  a comprehensive  geo- 
graphic base  file. 

Some  examples  of  geographic  coding  applications 
are: 

— Assigning  school  district  numbers  to  census 
block  data  in  order  to  assess  school  districting 
needs. 

— Assigning  grid  coordinates  to  police  complaint 
records  so  computer  produced  maps  can  be  used 
to  help  deploy  police. 

— Assigning  census  tract  and  block  numbers  to 
health  records  to  allow  rapid  tabulation  and 
display  of  morbidity  and  mortality  rates  for 
small  areas  of  a city  to  assist  in  predicting 
health  facility  needs. 

Feasibility  Surveys 

Addresses  of  a subsample  of  the  25  percent  sample 
household  in  the  pretest  census  for  the  New  Haven 
SMSA  were  selected  for  two  mail-out/mail-back  sam- 
ple surveys:  The  New  Haven  area  travel  survey  (an 
origin  and  destination-type  survey)  and  the  New 
Haven  family  health  survey.  The  questionnaire  word- 
ing, format,  and  data-collection  procedures  for  these 
surveys  have  been  made  more  compatible  with  Census 
Bureau  procedures  so  that  the  sample  survey  data  will 
be  more  compatible  with  census  base  data.  The  Bureau 
of  Public  Roads  plans  to  arrange  for  such  surveys  in 
several  SMSA’s  after  1970,  using  sample  households 
determined  by  census  and  then  relate  the  travel  data 
to  journey- to- work  and  other  basic  census  character- 
istics. 

The  family  health  survey,  which  contains  questions 
pertaining  to  the  health  of  all  family  members,  child 
day-care  arrangements  for  working  mothers,  and  fam- 
ily planning  questions,  is  being  conducted  for  the 
maternal  and  child  health  section  of  the  State  of  Con- 
necticut. The  survey  is  sponsored  by  the  Connecticut 
State  Health  Department  through  funds  made  avail- 
able from  the  Children’s  Bureau.  Maternal  and  child 
health  officials  hope  to  determine  how  such  studies 
might,  when  combined  with  decennial  census  data, 
help  them  in  directing  their  funds  and  efforts  to  assist 
families  and  communities  by  providing  more  and  better 
child  day-care  and  health  centers. 

The  Children’s  Bureau  is  interested  in  repeating  the 
family  health  survey,  with  some  modifications,  in  20 
cities  across  the  Nation  in  the  fall  following  the  1970 


Census  to  assist  in  determining  sites  for  child-care 
facilities. 

Cooperating  Agencies 

In  addition  to  those  previously  mentioned,  the  fol- 
lowing State  and  local  agencies  are  providing  assistance 
in  the  form  of  consultants,  staff  services,  equipment 
and  supplies,  office  space,  computer  and  reproduction 
facilities,  etc. : 

City  of  New  Haven, 

City  of  West  Haven  and  other  city  and  town 
planning  departments  in  the  New  Haven 
SMSA, 

South  Central  Connecticut  Regional  Planning 
Program, 

Community  Council  of  Greater  New  Haven, 
Community  Progress,  Inc.,  of  New  Haven, 

New  Haven  Redevelopment  Agency, 

Yale  University, 

Harvard  University, 

University  of  Connecticut, 

University  of  Southern  Connecticut, 

Connecticut  State  Government  Agencies: 

State  Highway  Department, 

Department  of  Public  Works, 

State  Health  Department. 

Considerable  assistance  has  also  been  provided  by 
local  business  and  industry  in  the  New  Haven  area, 
particularly  the  utility  companies. 

Special  Projects  and  Activities 

A few  of  the  projects  and  peripheral  activities  as- 
sociated with  the  New  Haven  census  use  study  are: 

1.  Yale  University  under  contract  to  NIMH  con- 
ducted a sample  survey  in  the  “hill”  project  area  of 
New  Haven  last  summer  and  will  provide  us  with 
addresses  of  their  sample  units  for  which  they  will  re- 
quest aggregated  census  tabulations.  The  base  for  the 
sample  was  Census  Bureau  updated  housing  unit 
block  counts. 

2.  Arrangements  are  complete  with  NIMH  for  an 
individual  matching  techniques  study.  Psychiatric  clin- 
ics in  and  around  New  Haven  will  record  the  neces- 
sary address  identifiers  and  personal  characteristics  of 
incoming  patients  for  a 6-month  period  ending  this 
summer  to  facilitate  a match  to  census  data.  Aggre- 
gated tabulations  on  families  of  the  patients  will  then 
be  provided  by  census. 

3.  The  Law  Enforcement  Assistance  Agency  of  the 
Justice  Department  has  provided  funds  to  the  city 
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police  department  under  a grant  to  explore  the  use 
of  computers  and  census  data  for  improved  national 
law  enforcement.  The  city  police  department  has  a 
copy  of  the  census  address  coding  guide  and  since  last 
summer  have  been  using  it  to  geographically  code 
police  records.  A matching  study  and  special  tabula- 
tions are  also  part  of  the  police  department’s  program 
under  the  grant.  A major  redeployment  of  police  per- 
sonnel took  place  in  March  as  a result  of  special  census 
data  and  other  police  department  studies  relating  to 
complaint  and  arrest  information. 

4.  The  Harvard  Laboratory  of  Computer  Graphics 
is  planning  to  attach  a staff  member  to  the  use  study 
office  to  use  the  census  pretest  data  to  produce  maps 
showing  housing  and  related  characteristics  under  a 
grant  from  HUD. 

5.  The  use  study  staff  are  cooperating  with  repre- 
sentatives of  the  Regional  Scientific  Center  in  New 
York  on  a project  to  display  census  data  on  their  geo- 
space plotter  using  the  DINE  system. 

6.  The  National  Center  for  Health  Statistics  plans 
to  compare  computer  coding  of  death  records  to 
clerical  coding  using  methods  developed  with  New 
Haven  vital  records  by  the  census  use  study. 

7.  Under  funds  made  available  from  the  Office  of 
Civil  Defense,  the  South  Central  Connecticut  Regional 
Planning  Agency  is  conducting  a community  shelters 
program.  Special  tabulations  have  been  requested 
which  will  assist  in  daytime-nighttime  population  esti- 
mates for  allocating  the  population  to  available  fallout 
shelter  space.  The  Office  of  Civil  Defense  has  asked  the 
use  study  to  study  the  problem  of  allocating  the  popula- 
tion to  shelter  spaces  by  computer  using  the  DIME 
system. 

8.  A project  sponsored  by  the  Children’s  Bureau 
and  the  Connecticut  State  Health  Department  will 
combine  data  from  the  State  birth  records  with  the 
obstetrics  information  from  the  Yale-New  Haven  Hos- 
pital and  summary  census  data  of  the  mother’s  resi- 
dence at  the  time  of  the  infant’s  birth.  These  data  will 
be  displayed  using  computer  graphic  techniques  and 
target  neighborhoods  will  be  defined  based  on  the  dis- 
tribution of  the  combined  local  and  census  data. 

These  are  only  a few  of  the  projects  being  developed 
at  the  use  study. 


Dissemination  of  Study  Results 

Results  of  the  use  study  will  be  considered  when 
planning  the  1970  censuses  and  other  census  programs, 
and  hopefully  by  local  communities  in  applying  and 
relating  the  results  of  the  census  to  their  own  data 
needs.  Success  of  the  study  will  depend,  therefore,  on  a 
fairly  extensive  program  of  education,  communica- 
tions, and  technical  aid  to  the  data  users.  For  this  reason 
both  written  and  visual  materials  will  be  prepared 
which  can  be  used  to  explain  the  study  results  and  rec- 
ommendations to  interested  groups  in  all  parts  of  the 
country.  Computer  programs  and  complete  “pack- 
ages” will  be  developed  for  use  by  local  planners  on 
local  computers. 

Brochures,  reports,  monographs  will  be  released 
periodically  beginning  this  summer. 

Summary  Tape  Workshops 

The  Census  Bureau’s  newly  created  Data  Access  and 
Use  Laboratory,  under  the  direction  of  Mr.  John 
Beresford,  is  establishing  contact  with  people  across 
the  country  who  would  like  to  act  as  Census  Summary 
Tape  User  Workshop  sponsors  after  the  1970  census. 
These  sponsors  would  set  up  workshops  to  show  people 
how  to  use  1970  census  summary  data  on  tape.  Sys- 
tems Development  Corp.  is  planning  to  work  with  the 
Bureau  in  assessing  the  technical  problems  in  the  areas 
of  tape  compatibility  between  census  and  noncensus 
hardware  systems,  software  packages  for  tape  use, 
urban  problems,  census  tabulation  uses,  urban  infor- 
mation systems,  etc.  If  anyone  is  interested  in  working 
with  census  summary  tapes  they  should  contact  Mr. 
Beresford  in  care  of  the  Census  Bureau  in  Washington. 

Plans  are  now  being  formed  to  have  the  workshop 
sponsors  come  to  New  Haven  beginning  next  summer 
for  census  summary  tape  seminars  which  would  take 
advantage  of  the  New  Haven  experience  and  would 
last  for  several  weeks. 

With  the  experience  of  the  New  Haven  census  use 
study  and  the  national  network  of  Census  Summary 
Tape  Workshop  sponsors  coordinated  by  the  Data 
Access  and  Use  Laboratory  and  Systems  Development 
Corp.,  plus  numerous  other  advances  in  the  areas  of 
data  collection,  processing  and  publication,  made  by 
various  divisions  of  the  Bureau  we  will  be  much  better 
prepared  to  serve  data  users  at  all  levels  after  the  1970 
census. 
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REPORT  ON  CURRENT  EFFORTS  TO  MATCH  PATIENT  INFORMATION 

TO  U.S.  CENSUS  RECORDS 

Richard  W.  Redick,  Ph.  D. 

National  Institute  of  Mental  Health 


The  Biometry  Branch  is  presently  engaged  in  carry- 
ing out  a census  matching  study  in  the  New  Haven 
metropolitan  area  in  connection  with  the  U.S.  Census 
pretest  conducted  there  in  April  1967. 

This  is  an  area  of  research  which  involves  the  utiliza- 
tion of  census  records  as  a source  of  data  for  studies 
designed  to  measure  the  rate  of  occurrence  of  a par- 
ticular event — in  the  present  case,  the  rate  of  admission 
to  psychiatric  facilities — among  specific  population 
groups  defined  by  various  demographic  and  socioeco- 
nomic characteristics. 

The  procedure  involves,  first  of  all,  identifying  indi- 
viduals to  whom  the  event  has  occurred,  in  our  case 
patients  admitted  to  psychiatric  facilities.  Secondly, 
locating  for  these  individuals  their  census  schedule 
which  contains  data  about  the  individual  and  the 
household  where  he  resides.  Third,  abstracting  from 
this  schedule  detailed  data  on  the  characteristics  of 
the  individual,  tabulating  the  data,  and  finally  relating 
this  numerator  or  patient  data  to  base  population  data, 
prepared  by  the  Census  Bureau,  in  order  to  obtain 
admission  rates. 

If  the  census  records  for  all  persons  included  in 
the  study  are  found,  it  may  be  assumed,  first  of  all, 
that  each  individual  counted  in  the  numerator  has 
been  enumerated  in  the  population,  and  secondly  that 
each  individual  is  classified  identically  in  both  the 
numerator  and  the  population  denominator  with  re- 
spect to  the  characteristics  under  study.  It  should  be 
pointed  out,  however,  that  these  assumptions  usually 
do  not  hold  if  the  numerator  data  are  obtained  from 
sources  other  than  census  records,  such  as  hospital 
records,  vital  statistics  records  or  patient  interviews. 

As  we  learned  from  a similar  study  carried  out  in 
1960  involving  the  use  of  census  records,  there  are 
problems  connected  with  the  use  of  this  census  ap- 
proach which  hamper  the  analysis  of  the  data.  These 
problems  must  be  minimized  if  this  approach  is  to  be 
used  in  any  future  studies  that  may  be  conducted  in 
connection  with  the  1970  census. 


In  this  context  then,  I want  to  briefly  describe  to 
you  some  of  the  problems  that  we  encountered  in  the 
1960  study,  and  indicate  how  the  study  that  we  are 
presently  conducting  in  New  Haven  is  attempting  to 
deal  with  some  of  these  problems.  In  conclusion  I will 
indicate  the  criteria  that  we  feel  are  needed  to  deter- 
mine the  feasibility  of  carrying  out  similar  studies  in 
1970. 

The  1960  census  matching  study  was  carried  out  in 
two  States,  Maryland  and  Louisiana,  and  included 
all  patients  admitted  to  psychiatric  facilities  in  those 
two  States  during  the  1-year  period  subsequent  to  the 
1960  census. 

A major  problem  that  we  encountered  in  analyzing 
admission  rates  for  this  study  was  the  fact  that  we 
were  only  able  to  locate  census  records  for  65  percent 
of  the  patients.  This  was  a considerably  lower  match 
rate  than  we  expected  since  two  other  census  matching 
studies  which  were  conducted  concurrently  at  that 
time,  one  involving  mortality  statistics  and  one  on  juve- 
nile delinquents,  had  match  rates  ranging  between  75 
and  80  percent. 

The  failure  in  our  study  to  locate  census  records  for 
almost  one-third  of  the  patients  was  attributed  to 
three  major  causes:  (1)  inaccuracies  in  the  patient 
data  that  were  obtained  from  the  psychiatric  facility, 
such  as  inaccuracies  in  the  patient’s  name  or  in  the 
address  information  given  to  us;  (2)  clerical  errors  in 
the  matching  operation  which  was  carried  out  by 
Census  Bureau  personnel ; and  (3)  under-enumeration 
of  population  by  the  Census  Bureau. 

Unfortunately,  sufficient  data  were  not  available  to 
indicate  the  relative  influence  of  these  three  factors, 
and  this  raised  a serious  question  for  us  concerning  the 
validity  of  admission  rates  based  on  the  matched  cases 
only. 

When  the  numerator  data  were  augmented  by  the 
nonmatched  cases  whose  characteristics,  were  deter- 
mined from  interviews  with  a 10  percent  random 
sample  of  the  patients  included  in  the  1960  study,  a 
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second  problem  arose  in  that  the  interpretation  of 
differences  in  rates  was  complicated  by  the  fact  that 
the  completeness  of  the  denominator  data  was  in  ques- 
tion owing  to  census  under-enumeration. 

Unfortunately  no  provision  was  made  in  the  1960 
study  to  carry  out  any  sort  of  systematic  or  intensive 
investigation  of  reasons  for  failure  to  match,  and  the 
Census  Bureau  up  to  that  time  had  made  little  prog- 
ress in  evaluating  the  1960  census  results  with  respect 
to  underenumeration,  although  they  had  produced 
some  net  census  undercount  by  age,  sex,  and  color. 

Consequently,  our  efforts  in  the  1960  study  indi- 
cated some  of  the  problems  connected  with  using  the 
census  record  approach,  but  did  not  provide  us  with 
any  solutions  to  these  problems. 

This  brings  us  then  to  the  study  that  we  are  cur- 
rently conducting  in  the  New  Haven  area. 

The  availability  of  census-type  data  from  the  April, 
1967  census  pretest  in  New  Haven,  provided  us  with 
an  opportunity  to  design  a methodological  study  which 
would  have  several  purposes : namely,  ( 1 ) to  investi- 
gate ways  of  insuring  a higher  match  rate  than  we 
obtained  in  the  1960  study;  (2)  to  examine  the  non- 
match cases  in  considerably  more  detail  to  pinpoint 
reasons  for  failure  to  match;  and  (3)  to  attempt  to 
determine  the  extent  of  under-enumeration  among 
various  subgroups  of  the  New  Haven  population,  and 
how  such  information  might  be  used  to  develop  ad- 
justment factors  for  analyzing  data  in  studies  of  this 
type. 

The  New  Haven  study,  like  the  Maryland  and 
Louisiana  study,  involves  patients  admitted  to  the  psy- 
chiatric facilities  which  serve  the  New  Haven  metro- 
politan area.  Identifying  information  is  being  collected 
on  all  admissions  over  a 6-month  period  subsequent  to 
the  April  1967  pretest. 

In  addition  to  the  sample  of  abQut  1,500  patients, 
similar  information  on  1,000  persons  from  the  general 
population  will  be  available  from  a survey  that  was 
conducted  in  the  Hill  area  of  New  Haven  and  in  the 
suburb  of  West  Haven. 

This  will  provide  us  with  the  opportunity  to  exam- 
ine match  rates  for  patient  and  nonpatient  population 
which  in  turn  may  provide  us  with  some  additional 
clues  about  the  feasibility  of  using  this  census  record 
approach. 

In  order  to  attempt  to  achieve  a higher  match  rate 
in  the  New  Haven  study,  i.e.,  a higher  match  rate  than 
was  obtained  in  1960;  some  revisions  in  the  procedures 
used  in  1 960  have  been  instituted  in  the  present  study, 
and  also  some  new  procedures  are  being  introduced. 
Included  among  the  revisions  are  some  changes  in  the 


content  of  the  patient  identifying  information  being 
collected  from  the  psychiatric  facilities  such  as  obtain- 
ing more  detail  about  the  names  of  the  patient  e.g., 
the  maiden  name  for  female  patients,  and  any  aliases 
or  nicknames  by  which  the  patient  may  be  known,  and 
attempting  to  get  more  detailed  information  about 
the  composition  of  the  household  where  the  patient 
resided  at  the  time  this  census  was  taken. 

In  connection  with  the  first  phase  of  the  matching 
operation  which  will  be  a clerical  or  hand  operation 
similar  to  that  carried  out  by  the  Census  Bureau  in 
1960,  the  criteria  used  to  specify  the  conditions  for 
determining  whether  a case  is  matched  or  nonmatched 
have  been  expanded.  Thus,  more  leeway  is  being  given 
in  matching  with  respect  to  the  address  information 
and  certain  characteristics  of  the  patient,  such  as  age. 

Subsequent  to  the  clerical  match  operation  we  will 
be  experimenting  with  computer  matching.  This  will 
involve  using  only  the  address  of  the  patient  as  well 
as  his  age,  sex,  and  color,  and  certain  characteristics 
of  members  of  the  patient’s  household.  The  computer 
match  will  eliminate  the  need  to  use  the  name  of  the 
patient. 

Subsequent  to  completion  of  the  two  matching  oper- 
ations, both  of  which  will  cover  all  persons  included  in 
the  study,  a comparison  will  be  made  of  the  results  of 
the  two  procedures  not  only  to  determine  which 
method  produced  the  greater  proportion  of  matched 
cases  but  also  to  examine  the  number  of  cases  which 
did  not  fall  in  the  same  category  of  being  matched  or 
nonmatched  in  the  two  procedures.  These  latter  dis- 
crepancies will  be  further  investigated  to  see  if  they 
throw  any  light  on  reasons  for  failure  to  match.  This 
may  partcularly  point  up  the  problem  of  clerical  errors 
that  occur  in  a hand  match  operation. 

Along  similar  lines  a detailed  investigation  will  be 
made  of  all  nonmatched  cases  to  determine  the  reasons 
for  failure  to  match  and  whether  or  not  for  certain 
categories  of  nonmatches  a further  search  by  either 
method,  that  is,  clerical  or  computer  matching,  could 
produce  additional  matches. 

In  the  1960  study,  it  was  found  that  the  nonmatched 
cases  fell  roughly  into  three  major  categories.  First, 
those  where  the  address  was  not  found  at  all.  This 
could  have  been  due  to  the  fact  that  either  the  patient 
reported  a completely  erroneous  or  nonexistent  address 
on  our  patient  information  form  or  that  the  Census 
Bureau  had  not  enumerated  the  address  at  the  time 
of  the  census.  The  second  category  included  those 
cases  where  the  address  was  found  but  the  household 
was  not  found.  This  may  have  resulted  either  because 
the  information  received  from  the  psychiatric  facility 
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on  the  patient  and  the  members  of  his  household  was 
completely  erroneous,  or  the  fact  that  the  census  listed 
the  address  but  failed  to  enumerate  the  household  at 
that  address.  In  the  third  category'  were  those  cases 
where  the  household  was  found  but  the  patient  was 
not  listed  as  a member  of  the  household.  This  could 
occur  where  the  patient  had  reported  information  that 
he  was  a member  of  a particular  household,  but  was 
actually  in  an  institution  or  living  elsewhere  at  the 
time  the  census  was  taken,  or  where  due  to  census 
under-enumeration  the  patient  may  have  been  a lodger 
or  a roomer  in  a household  but  was  not  enumerated 
as  a member  of  that  household  by  the  household  head. 

The  present  study  will  investigate  these  two  latter 
categories  of  nonmatches,  i.e.,  “address  found,  house- 
hold not  found”  and  “household  found,  person  not 
found,”  and  introduce  additional  procedures  for  deter- 
mining whether  or  not  a household  and/or  the  patient 
can  be  located  elsewhere  in  the  census  records.  Where 
these  procedures  are  successful  in  locating  the  patient, 
the  cases  will  be  reclassified  as  matches.  Documentation 
will  be  made  of  the  number  of  these  nonmatched 
cases  which  are  so  reclassified,  the  amount  of  time  and 
effort  required  to  locate  these  additional  matches,  and 
the  reasons  for  failure  to  match  these  cases  initially. 

For  all  other  cases  which  remain  in  the  nonmatched 
category,  an  attempt  will  be  made  to  determine 
whether  the  failure  to  match  was  due  to  errors  in  the 
identifying  information  obtained  on  the  patient,  or  to 
under-enumeration  by  the  Census.  The  lack  of  this 
information  in  the  1960  study  posed  problems  in  con- 
nection with  assessing  the  validity  of  admission  rates 
based  on  matched  cases  only. 

Relative  to  another  problem  encountered  in  the  1960 
study,  namely,  the  effect  of  census  under-enumeration 


upon  the  completeness  of  the  denominator  data  used 
to  compute  rates,  we  are  hopeful  that  the  Census  Bu- 
reau will  conduct  a detailed  investigation  of  the  extent 
of  under-enumeration  among  various  subgroups  of  the 
population  in  the  New  Haven  census  pretest.  This 
would  then  give  us  some  idea  of  what  problems  may 
be  encountered  with  respect  to  under-enumeration  in 
the  1970  census. 

For  purposes  of  assessing  whether  future  studies  of 
this  type  are  feasible,  we  have  set  down  some  of  the 
criteria  which  we  feel  are  needed  to  determine  this 
feasibility.  First  among  these  criteria  is  the  kind  of 
match  rate  that  can  be  obtained  in  a study  of  this 
type.  We  are  hopeful  that  the  New  Haven  study  will 
produce  a higher  match  rate  than  that  obtained  in  the 
earlier  1960  study.  A second  criteria  involves  the  extent 
to  which  the  Census  Bureau  will  examine  the  whole 
area  of  under-enumeration  in  the  1970  census,  i.e.,  will 
they  conduct  more  extensive  examination  than  that 
which  they  carried  out  in  connection  with  the  1960 
census,  and  will  it  be  done  fairly  quickly  subsequent  to 
completion  of  the  1970  census? 

A third  criteria  concerns  the  extent  to  which  a dis- 
tinction can  be  made  between  errors  in  patient  infor- 
mation and  census  under-enumeration  as  causes  for 
failure  to  match,  and  then  utilizing  this  distinction, 
relate  it  to  characteristics  of  the  nonmatched  cases  for 
purposes  of  developing  methods  to  adjust  for  non- 
matches when  examining  differences  in  rates  between 
the  various  population  groups  under  observation. 

A fourth  and  final  criteria,  which  may  prove  to  be 
the  most  important  determinant  of  all,  involves  the 
cost  of  carrying  out  studies  of  this  type  and  the  avail- 
ability of  funds  to  meet  these  costs. 
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CONTENT  AND  BEHAVIOR:  A SOCIAL  AREA  STUDY 
OF  NEW  YORK  CITY’ 


Elmer  L.  Struening,  Ph.  D.,  Stanley  Lehmann  and  Judith  G.  Rabkin,  Ph.  D. 
Albert  Einstein  College  of  Medicine,  Yeshiva  University 


Whether  seeking  to  explain  or  treat  identified  pa- 
thology, researchers  and  practitioners  in  the  main- 
streams of  medicine,  psychiatry  and  psychoanalysis 
have  traditionally  regarded  the  individual  as  their  focal 
unit  of  study.  Following  the  medical  model,  illness  was 
conceptualized  as  a foreign  agent  that  existed  within 
the  patient,  under  his  skin.  Isolated  individuals  opposed 
this  view,  but  in  general  the  relationship  between  the 
individual  and  his  environment  was  largely  ignored. 

Within  the  past  several  years,  an  increasing  number 
of  writers  interested  in  behavior  and  behavior  disorders 
have  sought  to  modify  the  basic  assumptions  implicated 
in  the  clinical  or  medical  model  of  illness.  They  come 
from  such  diverse  backgrounds  as  psycho-analysis,  gen- 
eral systems  theory,  contract  psychology,  social  biology, 
and  ecology.  These  men  subscribe  either  explicitly  or 
by  implication  to  the  position  that  psychopathology  is 
not  “subdermal”  but  develops  in  a context  larger  than 
the  individual.  Instead  of  focusing  on  factors  within 
the  individual  that  lead  to  psychiatric  difficulty,  the 
subject  of  study  becomes  the  transactions  between  the 
individual  and  his  environment. 

In  the  earlier  stages  of  the  community  mental  health 
movement,  attention  was  largely  devoted  to  other  in- 
dividuals in  the  patient’s  environment,  but  there  has 
been  increasing  attention  directed  at  prevailing  envi- 
ronmental conditions  and  characteristics.  In  line  with 
this  trend,  mental  health  workers  have  become  inter- 
ested in  the  sociological  studies  of  urban  typology  and 
particularly  in  the  method  of  social  area  analysis  in 
relation  to  disordered  behavior  in  large  metropolitan 
areas. 

Recognition  that  the  social  context  might  have  equal 
or  greater  relevance  than  individual  characteristics  in 
predicting  behavior  has  led  to  a change  of  treatment 
strategy  as  well  as  focus.  Today  many  community  ac- 
tion programs  adopt  an  “advocate  role,”  fighting  for 
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the  individual  in  order  to  change  his  social  context, 
and  replacing  the  traditional  psychotherapeutic  goal 
of  only  changing  the  individual.  In  health  and  mental 
health  services,  the  clinical  or  medical  model  intro- 
duced to  19th-century  medicine  is  being  increasingly 
replaced  by  the  public  health  model.  The  latter  con- 
ceives of  illness  not  as  a collection  of  independent  and 
discrete  entities  like  measles  and  diptheria,  but  as  inter- 
changeable and  shifting  manifestations  of  disorder  best 
dealt  with  in  the  context  of  social  and  environmental 
factors. 

Despite  these  widespread  changes  in  theory  and  ac- 
tion characterizing  mental  health  and  other  community 
programs  across  the  Nation,  little  study  has  been  de- 
voted to  the  comparative  merits  of  the  various  methods 
of  intervention.  While  the  massive  influx  of  Federal 
aid  provided  under  the  Community  Mental  Health 
Centers  Act  of  1963  assures  widespread  acceptance  of 
the  public  health  model  in  psychiatry,  little  is  known 
about  the  techniques  that  will  most  effectively  fulfill 
current  goals. 

Many  mental  health  organizations  throughout  the 
country  will  soon  be  responsible,  for  the  first  time  in 
their  history,  for  the  mental  health  or  social  compe- 
tence of  entire  catchment  area  populations  rather  than 
just  those  who  actively  and  spontaneously  seek  treat- 
ment. These  changes  of  emphasis  and  structure  have 
profound  implications  for  the  planning  and  implemen- 
tation of  community  mental  health  programs,  and  also 
create  an  urgent  need  for  their  systematic  evaluation. 
Using  methods  derived  from  social  area  analysis,  we 
have  become  involved  in  the  problems  of  evaluation 
and  planning  of  such  programs,  as  well  as  the  analysis 
of  underlying  assumptions  regarding  the  effect  of  envi- 
ronment on  its  inhabitants. 

Specifically,  one  aim  of  the  present  study  was  to 
develop  a method  of  accurately  describing  the  social 
characteristics  of  demarcated  urban  areas  in  terms 
meaningful  to  the  planners  of  community'  health  pro- 
grams. A second  aim  was  to  evaluate  the  effect  of  en- 
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vironments  on  the  social  behavior  of  their  inhabitants, 
and  a third  was  to  develop  procedures  for  analyzing 
the  above  information  to  determine  the  most  effective 
means  of  intervention  and  to  compare  the  efficacy  of 
alternative  approaches  along  these  lines. 

Ideally,  an  integrated  and  comprehensive  informa- 
tion system  describing  the  catchment  area  population 
and  its  environment  should  be  available  as  one  basis 
for  planning  a network  of  mental  health  services.  This 
would  include  a series  of  population,  environmental 
and  behavioral  variables  and  their  interrelationships  to 
facilitate  the  design  and  implementation  of  community 
programs.  After  this  preliminary  phase  of  establishing 
an  information  system,  a research  project  should  be  in- 
augurated to  evaluate  the  effectiveness  of  existing 
programs.  This  would  entail  measurement  of  change  of 
variables  presumably  influenced  by  mental  health 
programs  with  'other  influences  held  constant.  The 
present  paper  describes  the  first  step  in  this  plan:  the 
establishment  of  an  information  system. 

The  set  of  procedures  we  designed  to  meet  these 
research  aims  is  largely  derived  from  work  in  urban 
sociology,  and  more  specifically,  social  area  analysis. 
Urban  sociology  is  a 20th-century  specialization  devel- 
oped by  Park  and  Burgess  at  the  University  of  Chicago, 
concerned  with  the  spatial  distribution  of  social  prob- 
lems and  the  formulation  of  urban  typologies.  Map- 
ping techniques  were  used  to  reveal  such  spatial 
distributions,  relying  largely  on  census  data  for  basic 
information.  The  early  theories  of  urban  typology  in- 
cluded Burgess’  concentric  zones  hypothesis,  Hoyt’s 
sector  hypothesis,  and  Harris’  multinucleation 
hypothesis.  Although  these  typologies  generated  con- 
siderable research,  flaws  in  underlying  assumptions 
precluded  their  validation,  and  interest  in  this  ap- 
proach declined.  More  recently,  the  studies  of  Shevky, 
Tryon,  and  others  in  the  analysis  of  social  areas  have 
provided  a theoretical  basis  and  specific  methods  for 
delineating  the  social  characteristics  of  areas.  These 
investigators  have  been  less  interested  in  spatial  distri- 
butions, and  define  social  structure  solely  in  terms  of 
the  relationships  among  characteristics. 

Shevky  and  his  colleagues  used  1940  census  data  for 
Los  Angeles  to  develop  three  indexes — social  rank, 
urbanization,  and  segregation — in  terms  of  which 
census  tracts  could  be  scored.  Tracts  with  similar  pat- 
terns of  scores  are  regarded  as  representing  a particular 
type  of  social  area.  Tryon,  working  from  1940  census 
data  for  San  Francisco,  derived  three  indexes  by  cluster 
analysis  of  census  items,  which  he  labelled  socioeco- 
nomic achievement,  family  life,  and  assimilation.  These 
are  roughly  equivalent  to  the  indexes  that  Shevky 


worked  out  on  theoretical  grounds.  Borgatta  and  Had- 
den did  a factor  analysis  of  32  census  tract  items  and 
obtained  three  stable  factors  that  applied  to  different 
regions  of  the  United  States:  socioeconomic  status, 
suburb  factor,  and  disorganization-deprivation.  The 
first  two  are  clearly  comparable  to  the  indexes  of 
Shevky  and  Tryon,  and  since  the  item  with  the  highest 
loading  on  disorganization-deprivation  is  the  percent 
of  Negroes,  this  factor  is  also  comparable  to  Shevky’s 
segregation  and  Tryon’s  assimilation  index. 

Social  area  analysis  has  been  adopted  by  psycholo- 
gists as  a method  for  studying  the  relation  between 
environmental  and  behavioral  variables.  Until  now,  it 
has  most  often  been  used  in  the  study  of  juvenile 
delinquency.  In  the  current  study,  we  decided  to  adopt 
its  general  framework  to  study  the  relations  between 
environment  and  variety  of  social,  psychiatric,  and 
physical  problems.  The  method  lends  itself  to  the  classi- 
fication and  manipulation  of  information  describing 
any  given  population  and  its  environment.  It  is  assumed 
that  any  large  geographical  area  can  be  divided  into 
two  or  more  subareas  to  be  observed  and  described 
just  as  people  or  objects  may  be  observed  and  described. 
These  geographical  subareas  or  units  ideally  have  the 
following  characteristics : 

1.  Homogeneity  in  terms  of  the  variables  observed 
so  that  their  measurement  will  accurately  describe 
the  area.  For  example,  a unit  may  be  delineated 
so  that  it  includes  very  poor  and  very  rich  people. 
If  one  is  interested  in  financial  status,  the  statistic 
of  median  income  in  such  an  area  does  not  accu- 
rately describe  the  population  contained  in  it. 

2.  Stable  boundaries  to  permit  the  measurement  of 
change  over  time  on  a given  set  of  measures.  Since 
census  data  is  often  used,  there  are  characteristic- 
ally 10-year  intervals  between  measurements. 

3.  A sufficiently  large  population  in  each  unit  of 
observation  to  yield  reliable  observations. 

There  must  also  be  enough  units  of  observation  to 
insure  stable  or  replicable  results  and  relationships. 
The  reliability  and  validity  of  the  measures  chosen  for 
study  should  be  relatively  invariant  across  the  units. 
Since  the  variables  they  are  intended  to  describe  may 
have  different  statistical  distributions,  both  simple 
and  complex  relationships  among  these  measures  must 
be  considered. 

In  the  course  of  reviewing  possible  sources  of  data 
for  New  York  City,  it  became  apparent  that  public  and 
private  agencies  use  many  different  units  of  observa- 
tion, often  based  on  heterogeneous  areas,  making  com- 
parative and  correlational  studies  difficult  if  not  im- 
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possible.  To  mitigate  the  effect  of  such  problems,  we 
chose  the  health  area,  as  defined  .by  the  New  York 
City  Department  of  Health,  as  our  unit  of  observa- 
tion. Since  each  health  area  is  composed  of  two  or 
three  census  tracts,  we  were  able  to  describe  each 
health  area  in  terms  of  the  indices  available  in  census 
data,  as  well  as  epidemiological  data  compiled  by 
New  York  City  agencies  that  are  based  on  health 
areas. 

We  chose  25  variables  describing  social  and  behav- 
ioral characteristics,  most  of  which  were  measured  at 
two  points  in  time.  This  study  reports  on  only  19  of 
them,  each  representing  a single  occasion.  In  order  to 
have  our  data  in  comparable  form,  we  used  frequency 
of  occurrence,  with  two  exceptions,  to  quantify  our 
variables;  this  was  calculated  separately  for  each  of 
the  health  areas.  We  classified  these  19  variables  into 
two  groups.  The  first,  derived  from  census  data,  was 
regarded  as  descriptive  of  the  environment,  and  in- 
cluded such  variables  as  unemployment,  overcrowding 
and  the  number  of  residents  who  lived  abroad  in  1955. 
The  latter  variable  was  highly  correlated  with  number 
of  Puerto  Ricans  in  the  population  while  also  includ- 
ing a migration  component.  The  other  group  of  vari- 
ables describes  the  behavior  health  and  mental  health 
of  populations,  including  number  of  premature  births 
and  number  of  psychiatric  hospitalizations.  The  latter 
group  of  measures  was  conceived  as  outcome  variables 
presumably  linked  to  environmental  conditions. 

Thus,  we  selected  19  variables  to  work  with  in- 
tensively; of  these,  nine  were  regarded  as  predictor 
variables  describing  environmental  conditions.  The 
other  10  were  regarded  as  outcome  variables  describing 
behavior,  health,  and  mental  health,  presumably  repre- 
senting certain  crucial  outcomes  of  living  determined 
to  some  extent  by  environmental  conditions. 

Environmental  (predictor)  variables 

From  1960  census  data  unless  otherwise  noted: 

1.  Number  of  unemployed  people  14  years  or  older. 

2.  Number  of  overcrowded  housing  units. 

3.  Number  of  people  living  in  the  South  in  1955 
(mostly  Negro) . 

4.  Number  of  people  living  abroad  in  1955  (mostly 
Puerto  Rican) . 

5.  Median  family  income. 

6.  Number  of  people  14  years  or  older  who  are 
divorced  or  separated. 

7.  Median  years  of  education  completed  by  people 
25  years  or  older. 

8.  Number  of  people  on  welfare  funds  (1962). 

9.  Number  of  people  in  the  health  area. 


Behavioral  (outcome)  variables 

From  1960  data  unless  otherwise  noted : 

1 . Number  of  admissions  to  State  and  city  mental  hos- 
pitals ( 1961 ) . 

2.  Number  of  out-of-wedlock  births  (1962). 

3.  Number  of  infant  deaths  under  1 year. 

4.  Number  of  births  under  2,501  grams  (premature) . 

5.  Number  of  births  with  prenatal  care  beginning  in 
ninth  month. 

6.  Number  of  births  with  prenatal  care  beginning  in 
seventh  or  eighth  month. 

7.  Number  of  cases  of  reported  venereal  disease. 

8.  Number  of  reported  homicides. 

9.  Number  of  reported  suicides. 

10.  Number  of  reported  arrests  in  the  age  group  7-20. 

Our  method  of  evaluation  was  to  study  the  data  for 
each  health  area  in  a borough  of  New  York  City,  with 
the  intention  of  studying  all  five  boroughs  separately 
and  ultimately  combining  data  across  boroughs.  Al- 
though our  “N”  is  the  number  of  health  areas  per 
borough,  the  data  is  based  on  all  of  its  residents.  In  this 
sense  we  are  not  using  samples  from  a larger  popula- 
tion, and  cannot  replicate  our  work.  However,  com- 
parison of  data  from  the  different  boroughs  and  at 
different  points  in  time  can  provide  some  estimate  of 
the  generality  and  stability  of  our  results. 

At  present,  we  have  completed  preliminary  analyses 
for  two  boroughs,  the  Bronx  and  Brooklyn.  The  Bronx 
contains  63  health  areas  with  an  average  population  of 
twenty-two  and  a half  thousand  and  a standard  devi- 
ation of  about  7,000.  Brooklyn  contains  113  health 
areas  more  or  less  equal  in  size  and  distribution  to  those 
of  the  Bronx. 

In  order  to  assess  the  relationship  among  the  nine 
predictor  variables  describing  health  area  environ- 
ments and  the  10  outcome  variables  listed  above,  a 19 
by  19  product  moment  correlation  matrix  was  com- 
puted for  each  borough.  To  estimate  the  degree  to 
which  differences  among  the  health  areas  could  be 
explained  by  a comprehensive  description  of  environ- 
mental conditions,  multiple  correlation  coefficients  be- 
tween the  nine  predictor  variables  and  each  of  the 
outcome  variables  were  computed.  Throughout,  the 
effect  of  population  size  of  the  health  area  was  held 
constant  by  subtracting  from  the  square  of  the  multiple 
correlation  coefficient  the  square  of  the  product  mo- 
ment coefficient  between  the  population  size  variable 
and  the  outcome  variable.  The  square  root  of  the  re- 
sulting value  is  the  health  area,  size-corrected  multiple 
correlation  coefficient,  indicating  the  degree  of  rela- 
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tionship  between  a weighted  linear  combination  of  the 
nine  predictor  variables  and  the  outcome  variable. 

Table  1 presents,  for  each  borough,  the  multiple 
correlation  coefficient  between  the  nine  predictor  vari- 
ables and  each  outcome  variable,  and  this  coefficient 
corrected  for  population  size.  It  can  be  seen  that  the 
degree  of  relationship  between  the  combined  predictor 
variables  and  the  outcome  variable  is  consistently  high 
for  both  boroughs,  except  for  the  measure  of  suicides. 
In  both  Bronx  and  Brooklyn  data,  six  of  the  10  cor- 
rected multiple  correlations  exceed  .90,  indicating  a 
strong  relationship  between  environmental  conditions 
and  behavior  and  health.  Thus,  in  predicting  such 
factors  as  premature  births,  infrequency  of  use  of 
medical  facilities,  venereal  disease,  out-of-wedlock 
births,  and  reported  arrests,  more  than  three-quarters 
of  the  variance  is  accounted  for  by  environmental  pa- 
rameters. High  if  less  spectacular  correlations  were  also 
found  between  number  of  homicides,  infant  deaths, 
hospitalizations  for  mental  illness,  and  environmental 
variables. 


T^ble  1. — The  prediction  oj  the  10  outcome 
variables  corrected  for  health  area  population  size 


R* 

Corrected  R 

Outcome  variable 

Bronx 

N-63 

Brook- 

lyn 

N-113 

Bronx 

N-63 

Brook- 

lyn 

N-113 

Hospitalization  for 
mental  illness 

91 

92 

77 

86 

Out-of-wedlock 

births 

97 

97 

97 

97 

Infant  deaths. 

81 

93 

80 

90 

Premature  births 

93 

91 

90 

84 

Prenatal  care  (9th 
month  only) 

95 

96 

95 

94 

Prenatal  care  (7th- 
8th  month)— 

97 

97 

97 

96 

Reported  V.D 

97 

96 

97 

95 

Reported  homicides.. 

68 

82 

68 

82 

Reported  suicides 

48 

66 

35 

55 

Reported  arrests 

94 

92 

94 

92 

* Decimals  omitted;  all  values  in  .XX  format. 


Relationships  among  the  nine  predictor  variables 
are  moderately  high.  For  the  Bronx  they  have  a mean 
correlation  of  .53,  and  for  Brooklyn,  .49.  This  suggests 
that  the  number  of  variables  in  our  predictor  battery 
could  be  reduced  without  substantially  reducing  its 
power.  This  will  be  studied  systematically  by  compar- 
ing the  efficiency  of  different  combinations  of  predic- 
tor variables  in  predicting  different  outcome  variables 


in  the  five  boroughs  of  New  York  City.  Based  on  data 
we  have  at  this  point  it  appears  that  the  role  of  a 
predictor  variable  varies  considerably  as  a function 
of  the  particular  outcome  variable  it  is  to  predict  and 
of  the  borough  in  which  it  was  observed. 

Description  of  a given  health  area  in  terms  of  the  19 
environmental  and  outcome  variables  has  considerable 
meaning  to  the  health  and  mental  health  planner.  The 
number  of  admissions  to  mental  hospitals  provides  one 
estimate  of  the  manpower  needed  to  contend  with  such 
related  problems  as  helping  former  mental  patients 
adjust  to  community  living.  The  number  of  pregnan- 
cies receiving  medical  care  only  in  the  third  trimester 
reflects  the  nature  of  attitudes  toward  the  use  of  health 
facilities  and  almost  certainly  the  quality  and  style  of 
service  available  to  these  people.  The  extent  of  migra- 
tion from  southern  rural  areas  where  medical  and 
social  services  were  seldom  available  offers  one  basis 
for  planning  educational  programs. 

Such  variables  also  describe  the  type  of  stimuli  which 
constantly  impinge  upon  the  people  of  an  area.  As 
Tryon  points  out,  even  if  an  individual  is  different  from 
the  majority  of  people  living  in  his  neighborhood  and 
is  therefore  inadequately  described  by  overall  statistics 
and  demographic  data,  we  still  have  specific  knowledge 
of  the  social  situation  in  which  he  lives.  As  we  have 
found,  description  of  environmental  variables  reveals 
the  level  of  social  competence,  nature  of  the  opportu- 
nity structure,  extent  of  antisocial  behavior,  and  family 
stability  characterizing  a given  area.  Certainly  all  of 
these  factors  play  a role  in  a resident’s  level  of  aspira- 
tion, self-esteem  and  pattern  of  behavior. 

The  question  of  the  generality  of  our  results  is  of 
great  interest  to  us.  In  view  of  the  variety  of  possible 
factors  influencing  the  observed  relationships  between 
predictor  and  outcome  variables,  the  similarity  of  the 
obtained  multiple  correlation  coefficients  for  the  two 
boroughs  is  striking.  Seven  of  the  10  uncorrected  R’s 
varied  no  more  than  two  points  between  boroughs. 
However,  we  cannot  be  sure  that  the  same  findings 
will  emerge  elsewhere,  although  we  accept  them  for 
Queens  and  possibly  Manhattan,  possibly  with  some 
shift  in  the  relative  weights  of  predictor  variables.  But, 
of  course,  this  is  an  empirical  matter  which  we  hope 
to  resolve  through  continued  study. 

Assuming  our  results  are  in  fact  general,  it  is  worth 
thinking  about  their  implications  for  planning  a com- 
munity health  program.  Is  it  possible  to  improve  the 
mental  health  of  people  living  in  miserable  physical 
and  social  settings  without  also  improving  those  en- 
vironmental conditions  to  which  behavior  appears  to 
be  so  powerfully  linked?  The  answer  to  this  question  is 
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probably  an  emphatic  “NO”  and  we  suspect  that  a 
number  of  community  mental  health  directors  are 
keenly  and  anxiously  aware  of  the  odds  against  them 
unless  the  more  fundamental  social  and  economic  prob- 
lems of  their  catchment  areas  are  also  considered.  By 
creating,  for  example,  the  Model  Cities  program,  the 
Federal  Government  is  indicating  the  need  for  com- 
prehensive services  focused  on  both  environmental 
conditions  and  their  consequences  in  order  to  eliminate 
slums. 

Whatever  may  be  the  best  strategy  for  improving 
the  functioning  of  communities,  our  main  interest  is  in 
the  development  of  a set  of  procedures  for  evaluating 
alternate  models  of  intervention.  The  20  multiple  cor- 
relation coefficients  presented  here  hardly  do  justice 
to  the  richness  of  the  data  at  our  disposal.  There  are 
three  additional  kinds  of  analyses  we  plan  to  undertake. 
The  first  concerns  the  classification  of  health  areas  into 
types  in  terms  of  their  similarity  on  both  predictor  and 
outcome  variables,  following  the  procedures  of  Tryon 
or  other  profile  analytic  methods.  With  similar  health 
areas  thus  identified,  changes  under  the  influence  of 
different  conditions  or  programs  could  then  be  com- 
pared at  regular  intervals  over  a period  of  time.  As 
mental  health  programs  become  activated  in  our  great 
cities,  changes  in  similar  social  areas  with  different  pro- 
grams could  be  compared  on  a variety  of  relevant 
predictor  and  outcome  variables.  Deviant  social  areas 
of  a city — those  changing  abruptly  in  one  way  or  an- 
other— could  be  identified  and  studied  to  determine 
those  factors  contributing  to  rapid  deterioration  or 
growth. 

A second  form  of  data  analysis  we  intend  to  pursue 
concerns  interborough  comparisons  in  order  to  deter- 
mine differential  effectiveness  of  predictor  variables 
and  the  social  significance  thus  entailed.  For  example, 
the  variable  showing  number  of  southern  Negroes  who 
came  to  New  York  City  within  the  last  5 years  is  a far 
more  powerful  predictor  of  social  and  health  problems 
in  the  Brooklyn  data  than  it  is  in  the  Bronx.  In  con- 
trast, recent  migration  from  Puerto  Rico  is  much  more 


strongly  related  to  maladaptive  behavior  in  the  Bronx, 
suggesting  that  the  Puerto  Rican  is  at  a greater  dis- 
advantage here  than  in  Brooklyn.  Quite  obviously,  more 
detailed  study  of  the  interrelations  between  minority 
group  membership  and  other  predictor  variables  is 
needed  before  reasonably  valid  interpretations  are 
drawn,  but  these  preliminary  observations  suggest  the 
value  of  this  kind  of  analysis. 

The  third  approach  to  data  analysis  that  we  are 
planning  concerns  the  relationship  between  our 
predictor  variables  and  a more  comprehensive  sample 
of  behavioral  measures  presumably  associated  with 
social  conditions.  The  range  of  measures  could  be  as 
broad  as  the  programs  they  were  to  evaluate.  For  ex- 
ample, by  describing  the  social  areas  of  a city  in  terms 
of  the  reading  levels  or  academic  growth  scores  of 
seventh  grade  students  living  in  each  social  area,  rela- 
tionships between  school  performance  and  the  neigh- 
borhood environment  could  be  estimated.  By  acquiring 
information  from  established  agencies  and  by  gathering 
additional  data  through  independent  studies,  each 
social  area  within  a city  could  be  comprehensively 
described. 

The  kind  of  data  needed  for  the  accurate  and  effi- 
cient description  of  urban  areas  is,  unfortunately,  often 
difficult  to  obtain.  It  seems  to  us,  however,  that  the 
amount  of  money  currently  spent  by  most  cities  in 
gathering  information  by  a variety  of  agencies  using 
different  and  incompatible  units  of  observation  is  prob- 
ably adequate  to  sponsor  a single  data-gathering 
agency.  Its  task  would  be  the  coordination  of  relevant 
information  for  effective  program  planning  and  evalua- 
tion. Such  an  agency  has  been  established  for  the  entire 
state  of  Maryland  so  that  it  would  seem  equally  feasible 
for  urban  use. 

While  the  foregoing  procedures  may  seem  needlessly 
complex,  we  submit  that  such  an  approach  is  necessary 
to  develop  an  empirically  based  urban  sociology  and 
to  successfully  evaluate  the  proliferating  attempts  at 
influencing  the  health  and  mental  health  of 
communities. 
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MEASUREMENT  OF  THE  IMPACT  OF  MENTAL  HEALTH  PROGRAMS 

Audrey  W.  Mertz,  M.D. 

Mental  Health  Division 
Hawaii  Department  of  Health 


I am  pleased  to  be  part  of  the  National  Conference 
on  Mental  Health  Statistics  this  year,  and  I hope  to 
learn  more  about  the  topics  listed  on  the  program — 
such  as  methods  for  measuring  need,  forecasting  utiliza- 
tion of  mental  health  resources  and  measuring  change 
in  patients.  These  topics  are  of  vital  concern  to  a 
program  administrator. 

I was  asked  to  speak  to  you  today  presenting  the 
point  of  view  of  a State  mental  health  program  admin- 
istrator on  evaluation  of  a mental  health  program.  I 
wish  I knew  more  about  the  subject  I am  to  speak 
on — the  measurement  of  the  impact  of  mental  health 
programs.  I like  that  word  “impact.”  There  are  many 
aspects  of  our  program  that  can  be  measured  but 
measuring  the  impact  is  the  payoff.  What  does  the 
program  accomplish?  What  difference  does  it  make 
that  the  program  exists  at  all? 

With  your  indulgence,  I would  like  to  review  my 
own  career  in  psychiatry  in  explaining  the  develop- 
ment of  my  interest  in  program  evaluation.  I began 
my  residency  training  in  psychiatry  in  the  traditional 
manner,  at  a State  hospital,  actually  the  Hawaii  State 
Hospital.  I was  assigned  a roster  of  patients  who  had 
been  the  responsibility  of  the  doctor  who  preceded  me, 
and  I started  getting  every  fourth  new  patient  coming 
into  the  hospital.  I began  accumulating  clinical  exper- 
ience in  a hurry,  especially  with  the  group  in  largest 
supply,  the  chronic  schizophrenics.  However,  manic 
patients  are  rare  and  I didn’t  know  much  about  them. 
My  first  manic  patient  was  a charming,  energetic 
little  old  lady — a retired  schoolteacher.  She  was  talka- 
tive and  persuasive  and  she  convinced  me  that  she 
needed  a day  pass  to  get  some  things  from  home  and 
settle  some  business.  In  those  days  the  doctor  was  the 
sole  authority  in  such  matters;  the  day  of  staff  team- 
work and  patient  government  had  not  arrived  to  add 
other  dimensions  to  the  evaluation  of  a patient’s  be- 
havior and  of  his  readiness  for  graded  levels  of  freedom 
and  responsibility.  So  I let  my  patient  go  out  in  her 
own  custody.  Ten  hours  later  she  was  brought  back 


by  the  police — singing  and  shouting  and  wrapped  in  a 
blanket.  She  had  visited  a series  of  friends  and  then 
moved  from  bar  to  bar  in  Waikiki.  From  there  she 
started  singing  and  dancing  and  disrobed  in  the  street. 
I was  astounded  that  a little  old  schoolteacher  would 
do  such  things!  I learned  the  value  of  objective  meas- 
urements of  patient  behavior. 

Later  I learned  the  intricacies  of  conducting  indi- 
vidual psychotherapy — how  to  understand  the  patient’s 
communication,  how  to  watch  the  patient’s  movements 
and  expressions,  how  to  listen  to  what  he  said,  how 
to  listen  with  the  “third  ear”  to  what  he  didn’t  say  or 
what  he  really  meant  to  say.  With  all  this  concentration 
on  technique  of  understanding  and  influencing,  it  is 
easy  to  forget  about  what  the  goals  of  treatment  are. 
The  patient  and  the  therapist  get  in  the  habit  of 
meeting  each  week  and  interacting  in  a certain  way. 
But  what  are  they  trying  to  accomplish  and  how  will 
they  know  when  they  have  accomplished  it?  It  is  good 
discipline  for  the  therapist  to  ask  himself  those  two 
questions  and  to  write  down  the  answers  in  his  first 
note  on  the  patient’s  chart — for  frequent  reference. 
What  is  the  goal  of  treatment,  and  how  will  we  know 
when  it  is  reached? 

After  my  period  of  training  I became  chief  of  a 
mental  health  clinic  with  outpatient  and  consultation 
services.  Then  I not  only  had  to  evaluate  my  own  work 
with  patients  but  also  to  make  decisions  for  the  clinic. 
These  decisions  involve  evaluation.  What  patients  are 
suitable  for  mental  health  services?  For  outpatient  care 
instead  of  inpatient  care?  Which  patients  could  be  as 
well-served  by  family  service  agencies,  school  giudance 
counselors,  clergymen,  or  family  doctors — so  that  the 
time  of  the  mental  health  clinic  staff  can  be  conserved 
and  given  as  consultation  instead  of  direct  care  to 
patients?  Which  patients  are  going  to  drop  out  after 
the  first  clinic  contact  or  are  not  going  to  keep  even 
the  first  appointment?  Such  patients  are  a drain  on 
clinic  time  and  cause  a lot  of  paper  work  with  nothing 
accomplished.  Should  such  unlikely  patients  be 
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screened  out  before  first  contact  or  should  the  clinic 
adapt  to  their  resistance  to  treatment  or  to  their  prob- 
lems in  getting  to  the  clinic  and  send  a staff  person  out 
to  talk  to  the  patient  where  he  is? 

These  questions  and  the  decisions  based  on  answers 
to  these  questions  are  considered  to  be  in  the  area  of 
clinical  judgment  and  are  of  importance  to  the 
clinician.  A clinician  seldom  has  the  opportunity  to 
get  followup  information  on  his  patients  after  his  con- 
tact with  them  ends.  Studies  of  the  status  of  clinic  pa- 
tients 1 year  and  5 years  after  termination  of  clinic 
contact  would  add  important  information  for  the  refin- 
ing of  clinical  judgment.  Such  studies  are  important 
also  in  measuring  the  impact  of  a mental  health  clinic 
in  its  direct  work  with  patients. 

Usually  the  professional  staff  do  not  have  time  to 
track  down  former  patients  and  evaluate  them.  How- 
ever, a simple  study  can  be  done  by  graduate  students 
or  other  people,  under  professional  direction.  They  can 
find  out  if  the  patient  had  further  psychiatric  symp- 
toms or  problems  after  the  clinic  service,  how  he  is 
functioning  on  the  job  and  in  his  family,  and  so  forth. 

When  the  clinic  becomes  oriented  to  serving  all  the 
people  in  a designated  catchment  area,  the  staff  have 
to  look  at  the  characteristics  of  the  people  who  use 
their  services  and  draw  conclusions  as  to  the  people 
who  don’t.  They  may  find  that  the  people  in  certain 
neighborhoods  use  the  service  minimally.  Is  that  be- 
cause they  don’t  know  about  the  service  or  can’t  get 
to  the  clinic  readily,  because  they  are  of  high  income 
and  use  private  practitioners,  or  because  they  are 
afraid  of  the  negative  attitudes  of  their  friends  and 
relatives  if  they  go  to  a place  for  crazy  people? 

Perhaps  the  clinic  never  gets  cases  under  age  5 
or  over  age  65.  Why  is  that?  Drug  abuse  by  high  school 
and  intermediate  school  children  is  a major  concern 
of  the  community,  and  yet  the  clinic  sees  few  of  these 
cases.  Why?  These  are  intriguing  questions  for  the 
administrator  who  takes  seriously  his  mandate  to  serve 
the  whole  population  in  his  area. 

How  can  these  questions  be  answered?  The  staff  can 
meet  with  the  agencies  they  look  to  as  referral  sources 
or  send  out  questionnaires  to  them.  They  can  try  to 
find  out  whether  their  services  are  well  known  and 
what  the  users  think  of  their  services.  This  will  give 
some  clues  as  to  why  certain  age  groups,  ethnic  groups, 
or  economic  groups  seldom  use  the  clinic  and  why 
other  groups  may  tend  to  overuse  the  clinic.  Potential 
users  of  the  service  can  be  asked  what  they  know 
about  it,  what  they  think  of  it,  and  why  they  do  or 
do  not  use  it.  Such  groups  of  potential  users  might 


be  residents  of  public  housing  projects,  parent-teacher 
associations,  and  unions. 

When  the  service  demands  far  exceed  the  clinic’s 
manpower,  the  administrator  is  forced  to  evaluate 
what  he  is  doing  and  to  decide  how  best  to  use  his 
limited  staff  hours. 

Most  likely  he  will  decide  to  curtail  direct  treatment 
of  patients  and  to  expand  indirect  treatment.  He  will 
encourage  other  helping  people  in  the  community, 
such  as  physicians  and  clergymen,  to  work  with  psy- 
chiatric patients  and  give  them  training  and  consulta- 
tion. Theoretically  he  will  thus  spread  the  influence 
of  the  mental  health  professionals  and  bring  help  to 
more  people.  But  evaluation  is  needed  to  determine 
whether  the  people  in  need  are  receiving  appropriate 
help  and  are  improving  in  some  measurable  way.  And 
again,  followup  studies  are  an  important  and  usually 
absent  aspect  of  determining  the  impact  of  a mental 
health  program. 

The  State  hospital  administrator  has  problems  like 
those  of  the  mental  health  clinic  director  but  is  usually 
less  able  to  control  the  input  of  patients  into  his  pro- 
gram. His  concern  is  doing  the  best  possible  job  with 
patients  sent  to  his  hospital  and  getting  them  out 
again  within  a reasonable  period  of  time.  Hopefully, 
he  has  some  staff  assigned  to  research  and  program 
evaluation  who  continually  study  the  effectiveness  of 
programs  within  the  hospital.  He  has  a measurement 
of  effectiveness  in  the  return  rate  of  those  sent  out. 
Were  they  sent  out  too  soon?  Did  they  actually  re- 
ceive the  aftercare  that  was  arranged  for  them?  What 
went  wrong  with  those  who  returned?  What  went 
right  with  those  who  did  not  return? 

When  I was  administrator  of  Hawaii  State  Hospital 
I had  two  sets  of  figures  I watched  closely — the  daily 
patient  census,  and  the  number  of  staff  vacancies.  And 
frequently  I looked  at  the  budget  figures — the  short- 
age of  money  for  purchase  of  medications,  for  activi- 
ties for  patients,  for  building  maintenance,  for 
furniture.  The  State  hospital  administrator  usually 
has  no  time  to  look  for  what  the  hospital  is  not  doing, 
that  the  community  needs  to  have  done,  or  to  look 
at  the  long-range  effects  of  the  hospital’s  program.  This 
kind  of  evaluation  is  best  done  by  people  outside  the 
hospital  staff. 

The  administrator  of  a statewide  mental  health 
program,  such  as  the  one  I am  in  charge  of  in  Hawaii, 
has  the  opportunity  to  see  how  the  pieces  of  the  total 
program  fit  together  and  has  some  power  to  bring 
them  into  a proper  fit.  With  a data  system  including  f 
all  parts  of  the  program,  such  as  we  have  in  the  Hawaii 
Mental  Health  Register,  it  is  possible  to  identify 
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1 problem  groups,  which  can  then  be  studied  in  other 
j | ways.  One  problem  group  are  the  patients  who  go  in 
and  out  of  the  State  hospital  several  or  more  times.  We 
need  to  learn  more  about  this  group  of  “revolving 
door”  patients  and  to  plan  better  for  them.  It  may  be 
that  a regularly  scheduled  balance  of  time  between 
Jj  home  and  hospital  is  best  for  a small  subgroup  of 
these  “revolving  door”  types.  Some  families  may  be 
able  to  take  care  of  a chronically  ill  relative  at  home 
most  of  the  year  but  need  to  take  their  vacation  away 
from  the  sick  person  and  put  him  in  the  hospital  one 
[ month  each  year.  Conversely,  a man  may  be  able  to 
take  his  chronically  ill  wife  home  for  weekends,  and 
during  his  vacation,  but  needs  to  have  her  cared  for 
in  the  hospital  during  his  work  week.  The  larger  group 
of  “revolving  door”  patients  could  probably  be  stabi- 
lized at  home,  or  in  a family  care  home  if  appropriate 
care  were  given  in  an  outpatient  or  day  hospital  serv- 
ice, with  the  capacity  to  give  extra  help  during  times 
of  crisis. 

Participation  in  mental  health  planning,  in  com- 
prehensive health  planning,  in  vocational  rehabilita- 
tion planning  and  in  planning  for  people  in  poverty 
areas  has  changed  our  view  of  what  our  purposes  and 
objectives  are  in  a mental  health  program.  This  change 
has  added  more  confusion  than  clarity.  It  has  erased 
some  of  the  boundary  lines  around  the  mental  health 
program  and  shaded  in  some  peripheral  areas  of  over- 
lapping responsibilities. 

For  example,  we  have  in  Hawaii  a newly  expanded 
and  coordinated  system  of  dealing  with  alcoholics.  We 
inspired  ourselves  to  take  more  responsibility  for 
alcoholism  by  following  the  court  cases  being  appealed 
in  various  parts  of  the  country  concerning  public 
drunkenness  offenders.  We  pioneered  in  Hawaii  by 
repealing  our  public  drunkenness  statue  and  adding 
I intoxicated  persons  to  the  provisions  of  our  new  mental 
health  law.  Without  any  additional  funds  at  the  outset, 
we  had  to  take  responsibility  for  the  cases  formerly 
charged  by  the  police  for  being  drunk  in  a public  place. 
We  brought  together  staff  from  the  private  general 
hospitals  willing  to  receive  these  cases  in  their  emer- 
gency rooms,  the  State  hospital,  the  mental  health 
centers,  the  social  services  staff  who  administer  the 
medical  care  program  for  indigents,  the  vocational 
rehabilitation  workers,  operators  of  private  rehabilita- 
tion facilities  for  alcoholics,  care  home  operators,  police 
officials,  and  others,  to  work  out  a coordinated  pro- 
gram under  the  leadership  of  the  Department  of 
Health. 

We  had  difficulty  in  giving  the  State  Legislature  a 
description  of  this  program  and  its  budgetary  needs. 


Each  of  the  State  agencies  involved  had  its  piece  of 
the  alcoholism  program  as  part  of  its  budget  request. 
There  was  no  established  way  of  indicating  the  total 
scope  and  coordination  of  an  interagency  program. 

The  same  type  of  joint  planning  and  coordination 
is  needed  in  other  areas  where  the  mental  health  pro- 
gram overlaps  other  programs. 

Most  of  the  preventive  programs  in  mental  health 
are  the  responsibility  of  other  agencies.  For  example, 
family  planning,  prenatal  care  and  well-child  care  are 
important  in  preventing  the  problems  that  children 
experience  when  they  are  brought  up  by  parents  who 
cannot  adequately  love,  protect,  train,  and  nurture 
them.  These  children  are  counted  among  the  slow 
learners  and  the  behavior  problems  and  later  among 
the  drop-outs  and  delinquents.  This  is  an  example  of  a 
maternal  and  child  health  program  that  has  conse- 
quences for  educational  and  law  enforcement  pro- 
grams. None  of  this  is  in  the  core  of  the  mental  health 
program;  it  is  not  the  direct  responsibility  of  the  mental 
health  program  administrator.  But  surely  it  is  part  of 
mental  health. 

In  the  new  mental  health  law  in  Hawaii,  which  we 
in  the  State  mental  health  program  had  a large  share 
in  writing,  we  gave  ourselves  the  authority  to  evaluate 
all  mental  health  program  activities  in  the  State — 
public  and  private.  So  far  we  have  not  done  anything 
about  evaluating  programs  that  we  do  not  operate  or 
help  to  finance.  The  Mental  Health  Association  comes 
forward  periodically  with  a stated  desire  to  evaluate 
mental  health  programs,  but  they  are  not  equipped  to 
do  this  without  our  help. 

The  most  crucial  review  of  our  State  mental  health 
program  is  done  annually  through  the  budgeting  proc- 
ess. We  are  scrutinized  by  the  executive  branch  and 
then  by  the  legislative  branch  of  State  government 
every  year.  For  about  8 years  the  executive  branch  in 
Hawaii  has  been  asking  the  departments  of  State  gov- 
ernment to  do  long-range  planning — -to  set  forth  objec- 
tives, to  relate  programs  to  objectives,  to  evaluate 
success  in  reaching  the  stated  objectives  and  to  indicate 
alternative  programs  that  might  be  used  to  reach  the 
stated  objectives.  An  information  system  is  needed  to 
produce  annually  the  data  required  to  show  whether 
objectives  have  been  reached. 

In  the  past  year  a program-planning-budgeting  sys- 
tem has  been  started  in  Hawaii  making  the  same 
requests  and  adding  requirements  for  cost  projections 
and  cost-benefit  analysis  of  the  various  alternatives. 
The  PPBS  material  is  an  aid  to  the  program  adminis- 
trator, to  the  budget  department  and  to  the  Governor 
in  carrying  out  the  functions  of  the  State  government 
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in  an  effective  manner.  It  is  also  an  aid  to  the  legisla- 
ture in  evaluating  the  executive  branch  and  in  making 
decisions  about  legislation  and  appropriations.  It  can 
and  should  lead  to  elimination  of  unproductive  or 
harmful  programs,  to  improvement  of  inadequate 
programs  and  to  trial  of  new  programs. 

I have  left  to  the  last  the  crucial  question  of  how 
to  set  the  objectives  of  a statewide  mental  health 
program.  In  Hawaii  we  have  an  unusual  situation  in 
that  all  tax-supported  mental  health  activities  are 
operated  at  the  State  level.  We  operate  nine  regional 
mental  health  centers,  the  State  hospital,  and  several 
specialized  programs.  The  administrators  of  these 
various  units  have  a little  flexibility  in  how  they  carry 
out  their  programs,  but  they  have  to  follow  the  guide- 
lines and  fit  into  the  planning  done  in  the  central 
office.  And  they  have  to  function  within  the  severely 
controlled  State  civil  service  and  fiscal  systems. 

We  usually  state  our  objectives  in  three  broad 
areas — patient  care,  training,  and  research.  The  main 
reason  for  our  existence  is  patient  care,  and  research 
is  to  learn  more  so  that  we  can  refine  our  methods  of 
patient  care.  In  the  past  few  decades  the  site  of  patient 
treatment  has  shifted  away  from  State  hospitals  toward 
a variety  of  community  facilities  and  programs.  State 
hospitals  have  vigorously  strived  to  avoid  a custodial 
function.  They  have  also  tried  to  avoid  a function  of 
character-building  for  people  with  antisocial  character 
disorders.  Mental  hospital  administrators  have  tried  to 
state  their  objectives  as  treating  and  rehabilitating  pa- 
tients with  psychotic  disorders  and  certain  other  psy- 
chiatric illnesses.  They  want  to  refuse  people  they 
consider  social  misfits  rather  than  psychiatric  patients, 
and  they  want  to  discharge  after  a while  patients  who 
are  not  improving,  saying  that  they  have  achieved 
maximal  hospital  benefit.  Yet,  courts  and  physicians 
want  to  send  to  the  hospital  miscellaneous  people  who 
cause  community  problems. 

So  the  issue  is — can  the  State  mental  health  pro- 
gram limit  its  objectives  to  those  things  it  wishes  to 
do  or  will  other  demands  be  made  of  it  regardless  of 
stated  objectives?  Negotiation  on  this  occurs  at  the 
political  level,  and  shifts  may  take  place  from  time  to 
time.  For  example,  the  mental  health  program  in 
Hawaii  may  be  requested  by  the  legislature  to  get 
active  in  the  prevention  and  management  of  drug 
abuse  by  juveniles  even  though  we  have  not  sought 
responsibility  for  this. 

Can  all  the  objectives  of  a State  mental  health 
program  be  stated  in  terms  of  measurable  outputs? 
We  have  a staff  group  working  on  that  now  in  Hawaii. 
We  were  surprised  and  distressed  to  realize  that  our 


thinking  is  almost  entirely  oriented  toward  mental 
illness  and  not  toward  mental  health.  We  want  to 
stamp  out  mental  illness  by  early  case  finding  and 
vigorous  treatment  and  rehabilitation  of  all  possible 
cases.  But  we  don’t  know  much  about  enhancing  men- 
tal health  and  fostering  the  ability  of  people  to  cope 
with  adversity.  We  are  not  sure  which  groups  are  at 
high  risk  of  developing  mental  illnesses  so  that  we  know 
where  to  concentrate  preventive  efforts. 

There  are  other  goals  in  a mental  health  program 
which  may  not  sound  very  worthy  and  are  not  on  a 
par  with  such  objectives  as  reducing  the  suicide  rate. 
These  goals  are  such  things  as  meeting  professional 
standards  and  qualifying  for  Federal  funds. 

It  is  highly  important  for  a State  hospital  to  gain 
and  retain  accreditation  by  the  Joint  Commission  on 
Accreditation  of  Hospitals.  So  the  budget  request  in- 
cludes items  that  the  hospital  administrator  would  not 
want  except  to  meet  the  demands  of  the  joint  com- 
mission. More  elaborate  fire  safety  equipment,  more 
elaborate  medical  record  indexing,  and  more  tidy 
housekeeping  may  contribute  little  to  the  hospital’s  ob- 
jectives for  patient  care,  training  and  research  and  yet 
they  have  to  be  obtained. 

Programs  are  shaped  by  Federal  funding  mecha- 
nisms. The  goal  of  a State  budgetary  request  may  be 
to  meet  Federal  requirements  so  that  grants  can  be 
received.  Of  course,  a Federal  grant  should  not  be 
sought  in  the  first  place  if  it  does  not  make  sense 
locally  and  fit  in  with  local  program  objectives.  But 
the  way  the  federally  supported  program  is  developed 
may  not  be  the  way  best  suited  to  local  conditions. 

Major  Federal  programs  such  as  the  community 
mental  health  centers  programs  arise  from  political 
forces  mixed  with  professional  opinions  and  pressures. 
The  elaboration  of  objectives  of  such  a program  comes 
later.  Even  now  we  do  not  have  statements  from  the 
National  Institute  of  Mental  Health  on  objectives  of 
the  community  mental  health  centers  program  in 
terms  of  quantifiable  results  expected. 

None  of  us  in  government  programs  wants  to  be 
evaluated  solely  on  what  we  put  down  on  paper  as  to 
our  objectives  and  our  progress  in  meeting  these  ob- 
jectives. We  still  want  to  show  and  tell — to  have  the 
Federal  consultants,  the  Governor,  the  legislators,  and 
the  public  see  our  facilities  and  programs  and  know 
that  we  are  earnestly  trying  to  do  the  good  work  we 
are  trained  to  do.  We  want  personal  relationships  with 
the  decisionmakers  so  that  we  have  some  chance  of 
being  in  favor  and  getting  support  and  encouragement. 
But  we  can’t  go  by  forever  on  charm  and  dedication 
alone;  we  have  to  be  able  to  show  a desirable  impact 
from  our  programs  or  we  should  not  be  in  business. 
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EVALUATION  METHODS  AND  DYNAMICS  IN  PROGRAM  PLANNING 

Harold  D.  Holder,  Ph.  D. 

North  Carolina  Department  of  Mental  Health 


Let  me  describe  a division  of  statistics  in  a State 
mental  health  department  I once  visited.  The  statis- 
ticians reported  that  program  managers  often  asked 
for  data  that  was  not  available;  data  was  being  col- 
lected on  the  basis  of  “that’s  the  way  we’ve  always 
done  it”  rather  than  on  any  other  basis;  managers,  it 
was  reported,  had  difficulty  making  their  request  for 
information  in  a quantitative  manner;  and  many  things 
were  done  which  did  not  seem  particularly  useful  or 
meaningful  to  the  statistician  who  did  them.  I wonder 
this  morning  if  this  sounds  familiar  to  anyone  here. 

I am  in  a division  of  research  in  which  my  fellow 
researchers  are  studying  the  behavior  of  animals.  They 
have  called  my  attention  to  the  behavior  of  even  the 
smallest  of  creatures,  the  single  cell  organism.  Consider, 
for  example,  the  parameciam.  It  collects  data  about 
food,  threat,  and  the  presence  of  other  creatures  or 
objects  in  its  own  way.  It  does  not  set  up  an  elaborate 
data  processing  capability;  all  data  is  necessary  to  the 
function  of  the  animal — there  is  no  unnecessary  effort. 
Consider  man  as  a socially  functioning  organism  on  a 
much  higher  plane.  Man  has  an  amazing  ability  to 
collect  and  process  information  as  he  interacts  with 
other  men.  He  seems  to  have  a natural  tendency  to 
collect  that  information  most  relevant  to  his  needs. 

FIGURE  1 


Organizations,  including  those  in  mental  health, 
however,  do  not  have  the  natural  potential  (a)  for 
always  knowing  what  information  is  needed  for  best 
operations  and  (b)  for  sorting  through  many  irrele- 
vant bits  of  data  to  get  to  nuggets  of  “information.” 

Managers  in  mental  health  organizations  have  as 
their  primary  output  decisions  about: 

( 1 ) Organizational  goals, 

(2)  Allocation  of  resources, 

(3)  Identification  of  alternatives  for  reaching  goals, 

(4)  Selection  of  the  alternatives  to  be  used, 

(5)  Assessment  of  organizational  effectiveness. 

Most  often  these  decisions  are  based  on  personal 
experiences,  the  observation  of  others,  and  tradition; 
rarely  are  they  based  on  data.  (See  figure  1.)  On  the 
other  hand,  mountains  of  data  are  available  to  mental 
health  professionals.  An  examination  of  these  data, 
however,  reveals  simultaneously  a concern  with  a broad 
range  of  problems  reflected  in  scattered  and  free- 
ranging  research  and  a fairly  narrow  concern  for  data 
collected  on  a regular  basis.  Laboratories  and  field 
investigations  have  produced  an  overwhelming  set  of 
figures  and  research  observations.  At  the  same  time, 
State  and  Federal  statistical  services  collect  epidemi- 
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ological  data  on  incidence  of  institutional  admissions, 
types  and  numbers  of  diagnoses,  related  socioeco- 
nomic profiles,  length  of  stay,  types  of  treatment,  etc. 

What  has  been  the  payoff?  Dr.  Howard  Davis, 
Head,  Applied  Research  Branch,  National  Institute 
of  Mental  Health,  in  a speech  entitled,  “After  Evalu- 
ation, What?”  notes,  “A  scanning  of  some  1,000  an- 
notated citations  dealing  with  evaluation  in  mental 
health  yielded  short  of  a dozen  references  which  could 
be  inferred  to  be  devoted  to  the  utilization  of  evalua- 
tion findings  or  change  processes  based  upon  effective- 
ness measures.”  This  is  a clear  identification  of  a lack 
of  vitality  in  the  activities  we  currently  call  evaluation. 

Sensitivity  to  the  uncertainties  of  administering 
mental  health  programs  has  been  all  too  rare.  In  short 
we  are  suffering  from  data  gluttony  and  information 
starvation.  “Program  personnel  just  don’t  use  our 
data”  is  a common  complaint. 

The  topic  for  this  presentation  concerns  evaluation 
and  planning.  We  feel  these  have  to  do  with  manage- 
ment information  needs — they  do!  But,  unfortunately, 
the  definitions  we  use  of  both  are  varied  and  often 
contradictory.  The  activities  identified  as  “evaluation 
and  planning”  too  often  have  little  to  do  with  our 
pressing  problems. 

Evaluation  and  planning  represent  efforts  to  im- 
prove our  abilities  to  meet  the  mental  health  needs  of 
the  population.  But  questions  of  “what  needs”  of 
“what  population”  through  “what  means”  are  not 
answered. 

I believe  that  to  make 

(a)  our  data  meaningful  to  our  managers, 

(b)  evaluation  and  planning  meaningful  to  men- 
tal health  organizations,  and 

(c)  our  organizations  meaningful  to  people  we 
need, 

( 1 ) new  ways  of  thinking  about  ourselves,  our 
problems,  and  our  information  needs,  and 

(2)  techniques  for  implementing  or  making 
new  notions  operational. 

Modern  systems  approaches  have  the  potential  to 
meet  both  conceptual  and  technique  needs.  Therefore, 

I would  like  first  to  review  some  basic  systems  concepts 
and  then  to  look  at  how  a systems  approach  can  be 
used  by  mental  health  organizations  for  planning  and 
evaluation. 

A “system”  is  a set  of  elements  or  activities  linked 
together  in  a purposeful  way.  A system  is  not  a single 
grouping  (an  “assemblage”)  or  a collection  in  which 
there  is  pattern  in  the  relationship  (a  “systematically 


arranged  assemblage”).  In  a “system”  purposeful- 
ness is  served  by  relationship  and  linkage. 

To  the  degree  that  one  can  say  the  collection  of 
activities,  programs  or  behavior  such  as  community 
clinics,  patient  treatment,  administrative  arrangements, 
and  learning  environments  created  within  the  domain 
of  mental  health  serve  a purpose  is  the  degree  to 
which  such  a phenomenon  can  be  described  as  a “sys- 
tem.” There  is  no  assumption  here  that  existing  pro- 
grams or  structures  are  by  definition  systems.  To  define 
any  mental  health  phenomenon  as  a “system”  one 
should  be  able  to  determine  its  boundaries,  the  rela- 
tionships making  up  the  system,  and  identify  its  input 
and  output. 

The  boundary  of  a system  is  the  containing  lines  or 
set  of  lines  within  which  the  system  operates,  or  where 
the  system  ends  and  the  environment  begins.  Input  is 
that  which  enters  from  outside  the  system,  or  its  envi- 
ronment, and  output  is  that  which  flows  from  the 
system  to  the  environment.  Goals  are  the  system’s  rea- 
sons for  existence  or  what  the  system  wants  to  accomp- 
lish. A goal  is  a desired  end  state  which  the  energies  of 
a system  are  directed  to  achieve. 

A way  of  identifying  mental  health  counterparts  to 
systems  concepts  can  be  shown  as  follows: 

System  Boundary  = People-To-Be-Served 

System  Goals  = Program  Objectives 

System  = Program  or  Activities 

System  Input  = Resources 

System  Output  = Population  Changes. 

Purposefulness  exists  in  systems  in  many  forms.  An 
electric  motor  is  a relatively  simple  system  with  a fixed 
relationship  between  electrical  energy  input  and  revo- 
lutions per  minute  output.  The  goal  of  this  system,  built 
rigidly  into  it,  is  to  transform  electrical  energy  into 
mechanical  energy.  Such  a system  is  goal-oriented.  A 
goal-oriented  system  is  one  that  has  no  potential  for 
changing  itself  or  its  goals.  It  is  closed  and  has  no  ability 
to  interact  in  any  meaningful  way  with  its  environment. 
This  can  best  be  illustrated  by  a torpedo  shot  through 
the  water  toward  a target.  If  the  target  is  missed,  the 
torpedo  continues  on  its  projected  path  and  could 
with  an  unlimited  source  of  energy,  continue  indefin- 
itely. The  goal  can  in  no  way  affect  the  path  of  the 
weapon  once  it  is  launched. 

A higher  order  and  more  complex  system  is  one 
which  is  goal-directed.  Such  a system  is  open  to  the 
environment  for  interaction  as  an  essential  aspect  of 
the  system’s  existence.  Compare  the  close,  goal-oriented 
torpedo  with  its  technologically  sophisticated  sister  the 
heat-seeking  missile.  The  latter  is  constructed  to  collect 
and  process  information,  in  this  case  signals  of  heat 
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intensity,  in  order  to  continuously  change  its  course  of 
flight  in  seeking  its  target,  a maneuvering  jet  aircraft. 

Such  a system  utilizes  the  principle  of  feedback  to 
maintain  its  vitality.  The  meaning  of  feedback  has  be- 
come clouded  and  ambiguous  in  popular  usage.  Feed- 
back represents  the  process  in  which  a system  utilizes 
information  about  its  current  behavior  to  guide  future 
behavior.  Key  words  are  “utilize”  and  guide,  not  re- 
sponse or  reaction.  Berrien  points  out : 

Without  the  concept  of  feedback,  it  would  be 
nearly  impossible  to  understand  how  systems  man- 
age to  survive  and  function  even  for  short  pe- 
riods ....  The  importance  of  feedback  derives 
from  the  fact  that  ...  it  copes  with  a wide  va- 
riety of  disturbances  that  would  otherwise  disturb 
the  steady  state  of  the  system. 

In  feedback-control  systems,  deviation  from  the  goals 
creates  information  which  is  used  to  direct  the  be- 
havior of  a system.  Feedback  is  not  a blind,  casual  re- 
action to  a force  or  stimulus.  How  the  principle  of 
feedback  works  in  a purposeful,  goal-directed  system  is 
shown  in  figure  2. 

Mental  health  organizations  are  designed  social  com- 
plexities and  can  also  be  viable  systems.  Feedback 
would  exist  in  the  very  structure  of  such  organizations. 
The  behavior  and  evolution  of  these  organizations 
would  be  dependent  upon  the  information  the  orga- 
nization has  about  population  needs,  available  re- 
sources and  means  to  meet  needs,  and  the  basis  for 


changing  and  modifying  organizational  behavior. 
Every  mental  health  professional  personally  uses  feed- 
back in  his  daily  work  in  relating  both  to  individuals 
and  the  organization  in  which  he  participates.  How- 
ever, when  feedback  exists  only  informally  or  inter- 
personally,  it  does  not  necessarily  constitute  purposeful 
evaluation  or  planning. 

In  a viable  system  “evaluation”  and  “planning”  are 
essential  parts  of  management  and  consequently  the 
most  desired  feedback  process  in  mental  health  systems. 
Evaluation  is  the  process  of  determining  whether  a sys- 
tem has  reached  or  is  directed  toward  its  goals  in  order 
that  necessary  modifications  can  be  accomplished. 
Planning  is  the  process  of  keeping  the  system  relevant 
to  the  population  to  be  served  by  defining  and  redefin- 
ing goals  and  identifying  organizational  alternatives  for 
reaching  these  goals. 

Evaluation  and  planning  are  based  on  the  special 
range  of  decisions  made  by  managers  about  system 
activities.  Without  identifiable  and  publicly  shared 
goals,  evaluation  and  planning  do  not  exist. 

Evaluation  can  be  called  internal  feedback,  in  which 
the  output  of  the  system  is  measured  or  compared  with 
the  desired  output.  Internal  feedback  loops  are  closed ; 
information  is  internal  to  the  system  itself  and  is  related 
to  output  through  goals.  Ability  to  change  the  goals 
for  the  system  is  not  primary  to  feedback  in  this  level. 
Assumptions  for  this  internal  feedback  are  necessarily 
made  that  the  output  of  the  system  (its  interaction 
with  its  environment)  creates  desired  or  valued  states 


FIGURE  2 

Flow  Diagram  of  a Viable  System 
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in  the  environment  in  which  the  system  is  situated.  For 
example,  consider  a program  of  inoculation  against 
smallpox  to  provide  immunization  in  a given  popula- 
tion. Feedback  compares  the  level  of  actual  immuniza- 
tion against  the  desired  level.  The  assumption  is  made 
that  the  desired  level  of  immunization  contributes  to  a 
state  of  health  in  the  population,  that  is,  a high  per- 
centage of  the  population  not  infected  with  smallpox. 

This  loop  is  illustrated  in  figure  3.  The  activities  of 
the  system  (inoculation)  produce  output  (a  protection 
level)  for  the  environment  (the  target  population). 
This  output  is  monitored  (a  count  of  numbers  inocu- 
lated and  resulting  level  of  protection)  and  this  data  is 
used  by  the  monitor  (those  responsible  for  the  pro- 
gram) to  regulate  the  activities  of  the  system  (to  in- 
crease or  decrease  the  amount  of,  or  alter  the  pattern 
of  inoculation)  according  to  the  energy  and  resources 
available  (vaccine,  personnel,  equipment,  and 
candidates) . 

Planning  is  the  external  feedback  loop.  This  loop  is 
concerned  with  the  relevancy  and  appropriateness  of 


the  system  and  its  output  to  the  environment  or  society 
in  which  it  participates.  Without  this  loop  the  mental 
health  system  would  lack  essential  openness  and  would 
be  limited  to  its  own  internally  produced  information. 
Through  external  feedback,  the  system  takes  readings 
on  the  states  of  the  environment  which  society  values 
and  desires.  The  differences  between  internal,  closed 
loops  and  external,  open  loops  are  illustrated  by  con- 
trasting a normal  heating  system  which  maintains  room 
temperature  at  a predetermined  level  and  an  environ- 
mental-monitoring system  which  determines  the  appro- 
priate temperature  level  for  the  various  periods  of  the 
day  with  a consideration  of  the  number  of  people  in 
the  room.  Such  a system  could  keep  records  of  how 
temperature  varies  and  therefore  learn  (make  adjust- 
ment over  time) . 

Adjustment  of  goals  are  the  primary  output  of  the 
external  loop  as  the  system  works  to  maintain  a match 
with  the  desired  state  in  the  social  environment.  In  the 
case  of  the  immunization,  this  loop  would  determine 
socially-valued  goals  of  personal  health  or  the  level  of 


FIGURE  3 

A Closed  Feedback  Loop 
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“disease-freeness.”  The  loop  would  identify  what 
diseases  are  most  unacceptable  and  what  protection 
level  for  what  percentage  of  the  population  is  desired 
and  translate  this  into  operational  goals  for  the  total 
system.  This  loop  would  also  check  the  validity  of  the 
assumption  that  inoculation  contributes  to  states  of 
disease-free  health.  Figure  4 illustrates  the  coupling 
of  an  external  feedback  loop  with  the  internal  loop. 

As  a general  observation,  it  is  rare  to  find  either 
internal  or  external  loops  formally  and  purposefully 
existing  in  mental  health  organizations.  Our  organiza- 
tions more  frequently  take  on  the  character  of  goal- 
oriented  systems  like  the  torpedo. 

One  major  reason  why  evaluation  and  planning 
often  do  not  exist  is  the  lack  of  relevancy  managers  see 
in  activities  often  labeled  as  such.  If  evaluation  and 
planning  do  not  exist  as  a central  part  of  management 
they  do  not  exist ; in  other  words,  the  essence  of  evalu- 
ation and  planning  as  a feedback  process  is  contained 
in  management,  not  outside  it. 


The  notion  of  a system  and  the  cycle  of  feedback  to 
assist  a system  in  purposeful  change  as  the  environment 
changes  is  necessary  in  understanding  both  planning 
and  evaluation.  Evaluation  and  planning  are  built  into 
a system  to  maintain  the  desired  balance  or  interaction 
between  the  environment  and  the  system  in  the  case  of 
mental  health  between  our  clients  and  their  communi- 
ties and  the  formal  social  organizations  of  which  men- 
tal health  activities  are  a part. 

We  now  have  a basis  for  thinking  about  systems. 
What  is  meant  by  a “systems  approach?”  What  are 
the  techniques?  Unfortunately,  such  things  as  com- 
puters, mathematical  systems,  and  flow  charts  are  the 
popular  manifestation  of  techniques.  They  are  actually 
tools — physical  or  conceptual  instruments  like  tables  of 
random  numbers.  Techniques  are  the  way  of  using 
these  tools  to  deal  with  problems.  Techniques  of  sys- 
tems analysis  are  the  ways  of  using  systems  tools. 

The  basic  techniques  of  steps  for  systems  analysis 
are  concerned  with  boundary  definition,  system  defini- 


FIGURE  4 

System  with  Internal  and  External 
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tion,  goal  definition,  decision  criteria,  and  system 
modification.  To  use  a systems  approach  to  organiza- 
tions would  involve  the  following  questions: 

(1)  What  is  the  boundary? 

(2)  What  are  the  goals? 

(3)  What  is  the  system? 

(4)  Is  the  system  headed  toward  its  goals? 

(5)  What  adjustments  of  (a)  the  system  and  (b) 
the  goals  are  necessary? 

I.  The  Boundary 

A boundary  is  the  population  actually  served  by  or 
related  to  the  human-serving  system  of  which  mental 
health  defines  itself  to  be  a part.  All  boundaries  are 
arbitrary  limits  and  exist  in  the  minds  of  their  creators 
or  definers.  Therefore,  boundary  analysis  is  to  establish 
the  working  limits  of  primary  concern  to  a mental 
health  program.  For  example,  the  working  boundary 
may  be  only  those  people  who  are  directly  treated  or 
the  population  of  people  directly  served  by  a mental 
health  program  and  other  agencies  with  which  mental 
health  has  primary  and  defined  linkage  such  as  public 
schools  or  welfare.  The  boundary  could  be  all  public 
and  private  organizations  which  relate  to  the  problems 
of  people  or  view  people  as  problems.  Thus  boundary 
analysis  is  to  define  the  most  general  level  of  concern 
for  a mental  health  program.  In  other  words,  what  is 
the  population  which  is  primary  in  the  working  ar- 
rangements of  the  mental  health  program? 

II.  Operational  Goals 

A goal  is  a desired  state  which  an  organization 
wishes  to  achieve.  Too  often  organizational  goals  are 
a mixture  of  personal  goals  and  / or  private,  nonstated 
goals.  As  such  they  cannot  provide  direction.  Goals  are 
not  always  concerned  with  organizational  survival  or 
growth,  for  many  service  organizations  lack  well- 
defined  social  purposes  but  exist  on  the  basis  of  poten- 
tial for  survival  and  ability  to  consume  existing  re- 
sources. Goals  exist  at  a variety  of  levels  within  an 
organization,  ranging  from  those  most  immediate  and 
short-term  to  those  most  universal  and  long-term. 
Goals,  while  varied,  must  not  conflict  or  be  independ- 
ent; rather  they  must  be  complementary  and  congruent. 
Goals  at  any  level  in  an  organization  must  be  related 
to  all  goals  at  higher  levels.  Goals  can  relate  to  each 
other  in  a variety  of  ways.  Figure  5 illustrates  how 
organizational  goals  at  lower  levels  (shown  by  higher 
numbers)  are  linked  to  goals  at  higher  levels  (shown 
by  lower  numbers)  with  all  goals  finked  to  the  gen- 
eral purpose  of  the  organization  ( level  1 ) . This  means 


that  any  goal  at  a given  level  should  be  logically  linked 
to  at  least  one  goal  at  a higher  level  such  that  a “goal 
tree”  is  created.  In  figure  5,  the  left  hand  branch  is 
the  more  “theoretical”  branch,  whereas  the  “mixed” 
branch  is  more  likely  in  real  life.  Many  other  combi- 
nations are,  of  course,  possible.  To  coordinate  energies 
toward  this  goal,  an  organization  requires  a way  to 
measure  this  goal.  Without  measurement  the  degree  of 
goal-directedness  or  undirectedness  cannot  be  deter- 
mined. Progress  toward  any  goal  is  subject  to  variation. 
Therefore,  we  must  know  or  measure  the  degree  of 
direction  toward  or  direction  away  from  goal  attain- 
ment. This  requires  a decision  rule  or  way  to  know 
an  acceptable  variation  away  from  the  goal.  Figure 
6 illustrates  a plot  of  a system  operating  with  accept- 
able variation.  The  bank  of  allowable  variation  may  be 
constant  or  it  may  narrow  such  that  the  amount  of 
acceptable  variation  away  from  the  goal  is  reduced 
over  time. 

A system  monitors  its  goals  through  data  about  ex- 
ception (unacceptable  deviation)  from  its  goal.  If 
deviation  is  detected  then  adjustments  must  be  made. 
A goal  to  be  operational  must  provide  ( 1 ) a way  of 
measuring  a desired  outcome  and  (2)  decision  rules 
(road  signs)  for  determining  closure  toward  the  goal. 

III.  The  System 

A system  is  made  up  of  elements  or  subsystems  finked 
together  by  input  and  output.  Therefore,  mental  health 
systems  are  sets  of  programs  or  activities  which  mental 
health  controls  or  of  which  mental  health  programs 
are  a part. 

Designed  systems  are  those  created  for  defined  pur- 
poses. Natural  systems  are  those  which  grow  and  evolve 
naturally  without  design  out  of  the  combination  of 
available  resources  and  activities.  Unfortunately  for 
applying  systems  analysis  in  a direct  manner  (Given  a 
goal  or  problem,  what  designed  system  is  needed?), 
we  must  take  as  given  existing  organizations  and  seek  to 
bring  about  meaningful  change.  This  is  something  like 
rebuilding  an  automobile  while  it  rolls  down  the  street. 

In  terms  of  information  systems,  we  are  at  a point 
of  evolution  in  moving  from  existing  styles  of  identify- 
ing, collecting,  and  processing  important  data  to  new- 
er more  meaningful  styles.  Information  needed  by 
mental  health  management  is  not  limited  to  goal 
measurement.  As  important  in  the  design  and  modi- 
fication of  a system  is  examining  the  alternatives. 
Given  a goal,  what  are  the  alternative  ways  for  reach- 
ing the  goal?  Figure  7 illustrates  the  kinds  of  informa- 
tion which  are  required  in  choosing  among  alterna- 
tives. Thus,  management  requires  information  about: 
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( 1 ) The  alternatives  for  reaching  goals, 

( 2 ) The  costs  of  each  alternative, 

(3)  The  possible  outcomes, 

(4)  The  payoffs  or  values  of  the  outcomes, 

(5)  The  possibilities  a probabilities  ( Pi j ) of  that 
the  ith  outcomes  will  occur  if  the  jth  alternative 
is  implemented. 

How  many  of  us  can  provide  such  information  to 
management  now? 

IV.  Modifications 

Evaluation  is  concerned  with  an  examination  of 
programs  as  exception  from  goals  is  detected.  Plan- 


ning is  concerned  with  examining  the  appropriateness 
of  existing  goals  (and  thus  programs)  over  time  for 
meeting  the  mental  health  needs  of  people. 

How  can  this  systems  approach  be  integrated  into 
a mental  health  organization?  Figure  8 illustrates  the 
integration  of  conceptual  and  theoretical  activities, 
research  and  testing  activities,  and  implementation 
and  application  activities  with  the  five  phases  given 
for  using  a systems  approach  to  planning  and  evalu- 
ation. The  intersection  of  an  activity  category  with 
a phase  presents  the  specific  tasks  which  might  logically 
occur  as  a result.  For  example,  theoretical-conceptual 
questions  about  system  definition  are  those  of  “what 


Figure  5 

ILLUSTRATION  OF  A VARIETY  OF  LINKAGES 
BETWEEN  GOALS  OF  DIFFERING  LEVELS 
IN  AN  ORGANIZATION 
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Figure  6 

CURVES  SHOWING  TWO  POSSIBLE  TYPES  OF  SYSTEM  EFFICIENCY 


Lower  Limit  of  Tolerated  Variation 


is  the  boundary  of  the  system”  and  “what  makes  up 
the  system?”  Those  tasks  shown  are  suggested  and  not 
necessarily  the  only  ones  possible.  The  matrix  does 
illustrate  the  dynamic  interplay  between  the  varied 
components  of  the  systems  model  for  planning  and 
evaluation. 

Essential  in  systems  planning  and  evaluation  is  de- 
cisionmaking. Decision  represents  the  control  and 
judgment  aspect  of  management.  Without  manage- 
ment decisions  there  is  no  planning  and  evaluation. 
Often  administrators  expect  evaluation  and  planning 
to  be  the  domain  of  research  or  of  evaluation  and 
planning  specialists.  But  researchers  and  specialists 
can  act  only  as  supporters,  designers,  mechanics,  or 
engineers;  they  are  not  the  primary  element.  On  the 
other  hand,  evaluation  and  planning  needs  informa- 
tion. Data  alone  have  no  meaning  with  decision  capa- 
bility. Since  feedback  and  thus  evaluation  and 
planning  are  cycles  and  are  not  single  isolated  tasks  in 
a system,  information  must  be  continuously  available. 
Thus  the  notion  of  a management  information  system 
which  collects,  processes,  analyzes,  and  disseminates 
data  to  managers  is  central  in  a mental  health  system. 


Research  and,  in  general,  all  data  collecting  mecha- 
nisms, to  be  a part  of  a system,  must  be  linked  directly 
with  the  information  needs  of  decisionmakers.  In  other 
words,  they  must  be  judged  according  to  the  utility 
of  the  information  each  generates  in  support  of  man- 
agement. A system  is  concerned  with  utilizing  rather 
than  generating  data,  with  management  rather  than 
investigation. 

Conclusion 

To  summarize,  a mental  health  organization  viewed 
as  a system  requires  a comprehensive  program  for  the 
acquisition,  dissemination,  interpretation,  and  storage 
of  relevant  information.  Such  information  is  of  three 
basic  types: 

(1)  External — concerned  with  the  environment  in 
v/hich  the  mental  health  system  operates  such 
as  the  population-to-be-served,  community 
profile  and  needs,  available  resources,  mental 
health  problems  and  community  structure  which 
affects  and  can  affect  mental  health  efforts. 

(2)  Intrasystem — concerned  with  the  behavior  of 
the  system,  the  nature  and  activities  of  mental 
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Figure  8. — Matrix  Illustrating  Utilization  Of  General  Systems  In  Mental  Health 
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health  and  the  people  it  serves,  facilities,  fiscal 
activities,  direction  toward  goals. 

(3)  Intersystem — concerned  with  the  interaction  of 
the  system  with  its  environment,  such  as  the 
impact  of  mental  health  efforts  on  society  and 
vice  versa,  the  linkages  between  mental  health 
and  other  service  agencies. 

All  which  has  been  said  implies:  If  we  adopt  a sys- 
tems approach  to  planning  and  evaluation  and  thus  to 
mental  health  organizations,  then  information  needs 
will  change  drastically.  The  role  and  place  of  the  men- 
tal health  statistician  will  also  change  drastically  and 
will  become  Wen  more  central  and  more  meaningful 
in  management. 

Postscript 

One  way  to  respond  to  this  presentation  is  to  say, 
“That’s  all-theoretical  and  not  very  applied.” 

Yes,  that  is  in  part  a valid  response.  All  of  us  feel 
under  pressure  to  get  quickly-applicable  tools  and  tech- 
niques. We  want  patient-rating  scales,  reporting  forms, 


and  research  designs.  In  our  haste  we  seize  on  such 
tools  and  techniques  with  little  understanding  of  their 
basic  assumptions  and  conceptual  underpinnings  and 
their  potential  relevance.  Therefore,  we  often  put  band- 
aids  on  serious  lacerations  and  send  the  body  back  to 
do  battle. 

Systems  analysis,  as  well  as  other  alternative  ap- 
proaches to  planning  and  evalaution,  will  mean  little 
to  nothing  in  our  organization  unless  the  organization 
internalizes,  understands,  and  appreciates  the  ap- 
proach. Unless  the  mental  health  professional  realizes 
the  potential  benefits  to  him  and  his  efforts  to  serve 
people,  our  research  and  statistical  tools  and  techniques 
are  destined  to  be  more  ritual  than  program  reality. 

One  temptation  for  any  systems  advocate  is  to  talk 
about  computer  simulation  and  perturbation  analysis, 
queing  theory,  linear  programing,  game  theory,  and 
decision  theory  as  very  sophisticated  techniques  of  sys- 
tems analysis  and  operations  research.  But  for  all  their 
glamor,  they  are  an  empty  exercise  unless  they  are  a 
part  of  management. 
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The  tools  for  measurement  of  needs  for  mental 
health  services  are  largely  dependent  on  the  way  needs 
for  mental  health  services  are  defined.  Thus  we  have  to 
ask  the  questions,  who  defines  the  needs  for  services? 
In  what  framework  are  the  needs  conceptualized,  medi- 
cal, social,  personal?  What  is  the  relationship  between 
needs  for  services  and  utilization  of  services?  These 
issues  cannot  be  side-stepped  because  they  have  direct 
relevance  to  the  methods  for  measuring  needs. 

To  gain  perspective  on  these  issues  I will  discuss: 

I.  Definitional  Problems  of  Need: 

a.  Mental  Illness  Concepts  and  Treatment 
Modalities ; 

b.  Phases  of  Developments  of  the  Person; 

c.  Utilization  of  Services. 

II.  Methods  for  Measuring  Needs  for  Mental  Health 
Services : 

a.  Reporting  of  Utilization  of  Mental  Health 
Services ; 

b.  Survey  Methods; 

c.  Areas  and  Behavior; 

d.  An  Ecological  Method. 

I.  DEFINITIONAL  PROBLEMS  OF  NEED 

a.  Mental  Illness  Concepts  and  Treatment 
Modalities 

About  a century  ago  there  was  an  air  of  optimism 
in  professional  medical  circles  about  the  advances  in 
medicine.  Psychiatry  as  an  offshoot  of  medicine  shared 
in  this  optimism.  Thus,  mental  illness  was  seen  in  the 
same  way  as  other  illnesses,  particularly  the  most  shin- 
ing examples,  the  infectious  diseases.  Mental  illness 
could  be  described  by  its  symptomatology,  its  course 
over  time  and  its  outcome  and  in  time  we  would 
discover  the  causes.  It  was  indeed  a very  promising 
direction.  Out  of  this  medical  viewpoint  came  the 
description  of  many  forms  of  mental  illness.  (7)  Over 
a period  of  about  half  a century  the  various  subforms 


of  schizophrenia  were  differentiated  and  described,  and 
currently  these  are  categorized  under  schizophrenic 
reaction.  A similar  development  occurred  with  the  de- 
scription of  the  affective  psychoses.  At  the  same  time 
major  progress  was  made  as  to  the  differentiation  of 
the  global  causes  of  the  illness.  The  organic  brain  syn- 
dromes were  separated  from  the  functional  psychoses. 
Furthermore,  within  the  group  of  the  organic  brain 
syndromes  differentiation  as  to  the  causes  became 
clearer,  such  as  infectious,  circulatory,  toxic,  neoplastic, 
degenerative,  etc.  A similar  differentiation  occurred  in 
the  nonpsychotic  conditions  with  the  description  of  the 
neuroses  and  the  personality  disorders.  Disappearing 
from  the  psychiatric  vocabulary  were  descriptions  such 
as  weak  willed  and  immoral  personality.  All  the  cate- 
gories shared  the  assumption  that  mental  illness  was 
something  within  the  individual  and  that  in  time 
treatment  methods  would  be  developed  which  would 
either  stop  the  progress  of  the  disease  or  modify  the 
biochemical  abnormality,  etc.,  in  the  same  sense  as 
arsenic,  bismuth,  and  penicillin  are  treatments  for 
syphilis,  and  electroshock  and  antidepressants  are 
treatment  for  the  affective  disorders.  A nonorganic 
conceptualization  of  mental  illness  was  espoused  by 
the  mental  hygiene  movement  early  in  this  century'. 
There  was  a strong  emphasis  on  the  prevention  of 
mental  illness  and  thus  on  methods  of  child-rearing 
and  habit  formation.  This  movement  found  opera- 
tional expression  in  the  child-guidance  clinics. 

Another  conceptualization  of  mental  disorder  was 
that  psychopathology  could  emerge  as  a function  of 
the  interaction  between  people.  Similarly  if  it  could 
emerge  this  way  then  also  treatment  methods  could 
be  designed  in  an  interactional  framework  to  stop 
symptoms  and  prevent  them  from  recurring.  Within 
this  general  framework  the  concepts  of  therapeutic 
milieu,  therapeutic  community,  milieu  therapy,  and 
some  forms  of  group  and  family  therapy  emerged. 
{2,3)  This  last  conceptualization  does  not  place  the 
illness  within  the  person  but  it  is  a function  of  the 
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interaction  between  persons  and  this,-  then,  is  logically 
an  emphasis  on  the  structuring  of  the  social  psychologi- 
cal milieu. 

Many  issues  are  still  unresolved.  Is  social  incompe- 
tence, failure  to  adapt  to  the  demands  of  society,  a 
form  of  mental  illness?  (4,  5)  Is  unhappiness,  lack  of 
satisfaction,  and  inability  to  cope  a form  of  mental 
illness?  It  is  true  that  many  mentally  ill  are  socially 
incompetent,  unhappy,  dissatisfied,  and  unable  to  cope. 
Many  indices  of  personal  and  social  functioning  can- 
not be  ignored,  which  we  will  discuss  in  methods  of 
measuring  needs  for  mental  health  services. 

b.  Phases  of  Development  of  a Person 

Many  theories  of  personality  have  always  stressed 
the  importance  of  early  phases  of  development  of  a 
person.  The  successful  integration  of  the  experiences 
of  living  in  one  phase  of  the  development  would  con- 
stitute a solid  baseline  for  meeting  the  stresses  in  sub- 
sequent phases  of  development.  Developmental  psy- 
chologists have  stressed  that  to  pass  from  one  phase  of 
development  to  another  is  frequently  accompanied  by 
a period  of  relative  disintegration.  Those  periods  of 
disintegration  are  sometimes  characterized  by  erratic, 
moody,  withdrawn,  hyperactive,  seemingly  nonpur- 
poseful behavior.  This  is  the  way  we  tend  to  describe, 
e.g.,  adolescence  as  a phase  of  development.  However, 
later  in  life  there  are  also  important  transitional  phases. 
At  each  phase  in  the  development  of  persons,  new 
levels  of  integration  occur.  It  is  still  unclear  at  what 
period  of  a person’s  life  he  achieves  his  maximum  level 
of  integration.  Some  people  seem  to  have  "a  progres- 
sively ascending  life  line,  and  others  seem  to  start 
descending  in  terms  of  level  of  integration.  In  this  cul- 
ture we  seem  to  be  able  to  do  better  with  mastery  of 
new  challenges  than  with  having  to  realize  limitations 
of  self  and  the  loss  of  attributes.  At  many  points  during 
the  life  of  a person,  important  crises  occur.  Crises  are 
periods  of  transition  accompanied  by  personal  disinte- 
gration. Bereavement  of  an  important  person  is  such  a 
crisis.  Losses  do  not  only  occur  in  the  form  of  death  of 
a person.  A severe  loss  may  be  the  loss  of  youth,  of 
vitality,  the  loss  of  a job,  the  inevitable  realization  that 
there  is  no  further  promotion  in  sight,  the  relative  loss 
of  being  bypassed,  etc.  (6,7,8) 

When  we  measure  at  any  time  the  needs  for  services 
within  the  general  population  we  can  expect  to  find  a 
large  number  of  people  who  are  at  that  particular 
point  in  the  period  of  transition,  in  a state  of  disinte- 
gration. Generally  those  periods  are  characterized  by  a 
number  of  signs  and  symptoms  which  easily  could  be 
stated  in  terms  of  mental  illness.  Generally,  however, 


the  person  would  not  consider  himself  mentally  ill  or 
emotionally  disturbed  and  neither  would  those  in  his 
immediate  environment.  The  resolution  of  these  con- 
flicts, of  these  crises  of  transition,  are  not  purely  a mat- 
ter of  happenstance.  Significant  persons  in  the  imme- 
diate environment  present  models  for  coping  with  these 
crises,  and  at  times  institutionalized  rituals,  such  as 
funerals,  initiation  rites,  are  time  tested  prescriptions 
for  coping.  Unsuccessful  resolutions  of  those  crises  may 
result  in  a drift  toward  the  psychiatric  care  system.  (9) 

c.  Utilization  of  Services 

The  relationship  between  the  fractured  bone  and 
medical  attention  for  the  condition  is  clear-cut.  Such 
a direct  relationship  does  not  exist  between  emotional 
problems  and  the  delivery  of  mental  health  services. 
Various  studies  have  demonstrated  that  there  are  many 
gatekeepers  in  society  caring  to  some  extent  for  the 
mentally  ill.  In  order  of  importance  they  are  ministers, 
nonpsychiatric  physicians,  psychiatrists,  etc.  (10,  11) 
Furthermore,  the  more  severely  mentally  disturbed  per- 
sons on  the  welfare  roles  are  cared  for  by  welfare  work- 
ers. (12)  Rarely  are  the  highly  trained  professionals  in- 
volved with  these  persons.  Thus  utilization  of  formal 
psychiatric  services  may  in  no  way  reflect  actual  po- 
tential consumer  needs.  In  some  instances  flagrant 
psychopathology  of  a person  is  not  defined  as  ill  but 
eccentric.  (13) 

Another  aspect  of  utilization  of  services  is  that  the 
services  are  frequently  not  known  and  the  known  serv- 
ices may  be  avoided  because  of  the  stigma  attached  to 
being  considered  mentally  ill,  or  incompetent.  Many 
facilities  have  learned  that,  over  time,  more  and  more 
persons  tend  to  come  on  their  own  and  with  minimal 
coercion,  and  this  is  frequently  in  sharp  contrast  with 
the  initial  period  of  service  delivery  of  a facility  at 
which  time  many  persons  were  brought  by  the  arm  of 
the  law. 

A totally  different  aspect  of  help-seeking  behavior 
can  be  found  among  the  “sophisticated.”  To  be  in 
psychoanalysis  or  psychotherapy  can  be  a mark  of  dis- 
tinction and  fashionable  in  some  affluent  circles.  (14) 

There  is  also  increasing  evidence  that  utilization  of 
outpatient  services  may  be  a function  of  life  styles. 
Some  patients  tend  to  use  inpatient  services  only  and 
when  assigned  to  outpatient  services,  regardless  of  kind, 
individual  therapy  or  day  hospital,  drop  out  in  the 
early  phases,  or  never  come.  Others  appear  to  be  per- 
petual consumers  of  services  of  any  kind.  This  phe- 
nomena is  not  just  related  to  socioeconomic  class  but 
to  the  appetite  for  psychiatric  services,  and  probably 
other  human  services.  (15) 
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The  separate  reporting  of  outpatient  and  inpatient 
statistics  has  obscured  these  phenomena.  Only  since 
attention  has  been  paid  to  the  construction  of  treat- 
ment centers  has  a tentative  typology  of  user  type 
emerged. 

The  preceding  discussion  about  concepts  of  illness, 
phasic  development  of  a person,  and  utilization  of 
services  demonstrates  many  potential  pitfalls  in  relat- 
ing measurement  of  mental  illness  to  need  for  mental 
health  services.  Thus,  with  a great  number  of  reserva- 
tions, we  can  now  move  to  the  second  part. 

II.  METHODS  OF  MEASURING  NEED  FOR 
MENTAL  HEALTH  SERVICES 

a.  Reporting  of  Utilization  of  Mental 
Health  Services 

It  is  not  surprising  that  only  those  hospitals  who 
depend  on  public  money  use  a reporting  system.  Thus 
the  data  obtained  reflect  rates  of  hospitalization  in 
public  facilities  by  age,  sex,  diagnostic  categories  with 
an  overreporting  for  patients  in  the  lower  socioeco- 
nomic class  and  those  with  severe  chronic  illnesses. 
There  is  still  not  a uniform  reporting  in  the  United 
States  which  includes  all  private  and  public  institu- 
tions. This  is  regrettable  because  the  reporting  system 
has  never  violated  the  rights  of  individuals.  In  addi- 
tion, more  information  is  available  in  the  public  and 
private  nonprofit  institutions.  The  more  widespread 
the  use  of  general  hospitals  for  psychiatric  services  be- 
comes, the  more  crucial  it  will  become  to  integrate 
information  available  from  various  sources. 

Magnificent  work  has  been  done  with  an  inadequate 
reporting  system  describing  utilization  of  public  hos- 
pital beds,  needs  for  the  future,  the  ever  increasing 
number  of  admissions,  the  reductions  in  hospitals  cen- 
sus starting  in  1954,  shifts  in  hospital  use  for  various 
age  groups,  diagnostic  categories,  etc.  (16)  There  is 
an  unfortunate  tradition  in  reporting  which  separates 
outpatient  from  inpatient  statistics.  This  artificial 
separation  reduces  the  value  of  both  reporting  sys- 
tems, particularly  now  that  outpatient  services  are 
more  varied  and  more  utilized.  The  few  case  registers 
in  operation  require  close  scrutiny  as  to  their  utility 
and  expense.  We  should  consider  the  possible  advan- 
tages and  disadvantages  of  less-expensive  methods, 
such  as  the  prevalence  of  1-day  treatment  of  veterans 
in  public  institutions,  and  the  utility  and  cost  of  1-day 
treatment  prevalence  study  versus  3 months  or  possible 
1 month  application  for  service  studies.  Each  method 
has  its  advantages  and  disadvantages  in  relation  to 
specific  goals,  but  these  studies  are  necessary  to  arrive 


at  the  most  economical  and  least  cumbersome  report- 
ing system.  The  data  obtained  from  various  efforts  need 
to  be  examined  against  the  data  obtainable  from  a 
case  register.  Currently  some  efforts  have  been  started 
to  record  not  only  formal  psychiatric  services  but  to 
integrate  these  with  utilization  of  all  social  agency 
services.  These  projects  are  explanatory  in  nature,  but 
they  may  reveal  that  continuity  of  care  cannot  be 
achieved  as  long  as  we  accept  the  traditional  fragmen- 
tation of  health,  mental  health  and  social  services. 
Acceptance  of  the  fact  that  the  mental  health  service 
delivery  system  is  only  one  part  of  a complex  network 
of  services  might  lead  to  a more  comprehensive  and 
less  fragmented  system  of  services,  and  thus  also  of 
reporting  of  services.  If  this  is  the  direction  we  go  the 
reporting  system  may  very  well  come  much  closer  to 
being  a reflection  of  the  experienced  need  for  services. 

b.  Survey  Method 

This  is  an  established  method  and  has  been  used 
extensively  by  demographers,  human  ecologists,  social 
psychologists,  and  public  opinion  researchers.  The  size 
of  the  sample  depends  on  the  issue  to  be  clarified.  To 
illustrate,  I will  give  you  the  logic  of  one  survey.  The 
problem  we  tackled  is  the  prevalence  of  undiagnosed 
mental  illness  in  the  general  population.  From  infor- 
mation collected  on  persons  we  know  that  some  have 
subsequently  sought  help  or  had  recent  contact  with  the 
Western  Missouri  Mental  Health  Center.  It  is  esti- 
mated that  3,000  respondents  would  be  necessary  to 
produce  a cohort  of  50  contacts  within  1 year  ( 18  years 
of  age  and  over) . The  margin  of  error  is  plus  and 
minus  25,  so  that  the  range  of  the  cohort  would  be  ap- 
proximately between  25  and  75.  This  information  is 
based  on  comparative  register  data  in  Monroe  County, 
N.Y.  (17) 

Since  the  requirements  are  to  interview  only  persons 
1 8 years  old  and  over,  and  since  the  rates  of  completed 
interviews  runs  about  75  to  80  percent,  the  required 
sample  size  in  order  to  obtain  3,000  interviews  would 
be  4,000.  An  updated  city  directory  indicates  that  the 
area  we  intended  to  cover  contains  157,000  dwelling 
units.  This  means  a sampling  interval  of  39  to  obtain 
the  necessary  sample  size  of  4,000.  The  crucial  issue  is 
that  each  dwelling  unit  has  an  equal  chance  to  be 
selected  as  part  of  the  sample.  Thus,  if  nobody  is  home 
in  unit  39,  the  interviewer  cannot  go  to  unit  38  be- 
cause unit  38  had  a chance  to  be  selected  in  the  original 
sampling  decision.  Specific  procedures  are  used  to  ac- 
count for  the  “discovery”  of  new  dwelling  units  created 
after  the  city  directory  was  compiled,  and  thus  insure 
that  each  adult  in  the  household  has  an  equal  chance 


to  be  interviewed.  As  you  can  imagine,  there  are  many 
problems  in  finding  people  at  home  and  gaining  en- 
trance and  participation. 

Equally  important  issues  are:  What  to  ask?  How  to 
ask?  What  kind  of  “interviewer”  is  necessary  to  do  the 
job?  If  the  focus  is  to  determine  needs  for  mental 
health  services,  the  interviewer  should  be  able  to  elicit 
differentiation  between  a variety  of  conditions  such 
as  chronic  psychotic  conditions,  alcoholism,  crippling 
versus  mild  neurotic  conditions,  current  life  crises,  de- 
velopmental phase  problems,  etc.  These  problems  of 
definition  are  reflected  in  the  wide  range  of  rates  of 
mental  illness  found  in  various  studies.  A study  review- 
ing surveys  (18)  describes  rates  ranging  from  20  to  330 
per  thousand.  This  broad  range  has  led  to  greater  use 
of  more  qualified  interviewers,  even  psychiatrists.  (19) 
You  can  imagine  what  this  will  do  to  the  cost  of  a 
survey.  On  the  other  side  of  the  coin,  efforts  are  made 
to  develop  simple  test  instruments  with  power  to  dis- 
tinguish the  well  from  the  ill.  (20) 

Furthermore,  the  distinctions  being  made  should 
have  some  relevance  to  the  kind  of  treatment  needed. 
Some  recent  studies  indicate  that  the  decisionmaking 
processes  in  psychiatric  facilities  about  the  assignment 
to  treatment  modalities  is  far  from  simple  and  is  fre- 
quently less  related  to  the  state  of  psychopathology 
and  more  to  the  power  and  forcefulness  of  complain- 
ant as  well  as  past  treatment  history  of  the  patient. 
Thus  survey  methods  as  a tool  to  measure  need  for 
mental  health  services  suffer  from  many  imperfections. 

c.  Area  and  Behavior 

Problems  experienced  by  people  can  be  defined  as 
need  categories  such  as  depressions,  schizophrenia, 
school  dropouts,  criminal  behavior,  unemployment, 
etc.  The  question  is  about  their  relatedness.  Recent 
studies  have  focused  on  the  socioeconomic  class  and 
mental  illness,  and  kind  of  treatment  received  for  the 
mental  illness.  Attempts  are  currently  made  to  inte- 
grate delivery  of  a variety  of  human  services  such  as 
welfare,  health,  and  mental  health.  These  illustrations 
serve  to  indicate  broader  awareness  of  the  complexity 
in  the  delivery  of  mental  health  services. 

An  important  issue  is,  are  there  characteristics  of 
area  which  will  permit  a study  of  the  relationship  be- 
tween environmental  conditions  and  behavioral  mani- 
festations. Various  investigators  (21,  22,  23)  have 
taken  this  idea  seriously  and  have  developed  methods 
to  analyze  these  relationships.  This  approach  requires 
characterizations  of  environments  such  as  conditions 
of  housing,  characteristics  of  population,  living  ar- 


rangement in  those  areas.  Examples  of  behavior  char- 
acterizations are  crimes  committed  by  people  in  the 
area,  arrest  for  drunkenness,  help-seeking  behavior 
from  a mental  health  center,  receiving  public  assist- 
ance, ways  of  dying  (death  certificates),  library  use, 
school  dropouts,  completion  of  high  school,  aspiration 
level,  etc.  These  kinds  of  data  can  be  analyzed  cross- 
sectionally  and  longitudinally.  Theoretically  it  will  be- 
come possible  to  discover  a number  of  areas  which  are 
alike  in  some  ways  and  markedly  different  in  other 
ways,  e.g.,  Area  I and  Area  II  may  be  alike  on  numer- 
ous characteristics  except  two,  such  as  hospitalization 
rate  for  mental  illness,  and  dominant  family  structure. 
Such  a finding  could  then  lead  to  a comparative  study 
of  the  two  areas  focusing  on  the  relationship  between 
aspects  of  family  living  and  hospitalization  for  mental 
illness.  Furthermore,  other  analyses  on  area  character- 
istics may  demonstrate  the  total  unrelatedness  of  cer- 
tain behavioral  indices  and  conversely  the  clustering  of 
other  variables,  comprehensible  by  one  unifying 
concept. 

I foresee  many  practical  applications  of  these  kinds 
of  studies  such  as  generating  explanatory  concepts  re- 
lating mental  illness  to  environmental  conditions,  pro- 
viding guidelines  for  planning  and  measuring  the 
impact  of  intervention. 

d.  Ecological  Method 

Up  to  this  point  I have  discussed  data  collection  and 
analytical  approaches  which  deal  with  broad  categori- 
zations. The  next  necessary  step  is  an  approach  to  de- 
scribe with  precision  the  matrix  of  living  in  a small 
area,  for  example,  a block  in  the  city.  Only  when  we 
have  a more  detailed  account  of  the  way  life  is  being 
lived  can  we  hope  to  measure  impact  of  mental  health 
services  upon  the  quantitative  and  qualitative  elements 
of  the  matrix  of  living.  The  approach  is  well  developed 
for  rural  communities  and  will  be  modified  for  urban 
life.  (23,  24)  Against  the  background  of  knowledge 
about  normative  behavior  within  a small  area,  failure 
to  cope,  becoming  a patient  can  be  understood  better 
and  intervention  methods  can  be  tailored  for  individ- 
uals, families,  or  specific  target  population  in  a small 
area. 

Furthermore,  if  large  and  small  intervention  pro- 
grams, such  as  poverty  programs,  model  city  programs, 
GMMH  center  programs,  have  no  noticeable  impact 
on  the  quality  of  living  within  a study  area,  the  inevita- 
ble conclusion  has  to  be  that  the  stated  goals  are  not 
achieved  and  thus  alternatives  need  to  be  considered. 
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MEASURING  CHANGE  IN  PSYCHIATRIC  PATIENTS 

Allen  Raskin,  Ph.  D. 
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Today,  most  investigators  interested  in  evaluating 
the  effects  of  a treatment  procedure  will  rely  on  a 
rating  scale  to  measure  these  effects  in  psychiatric  pa- 
tients. Consequently,  the  main  thrust  of  this  paper  will 
be  on  psychiatric,  nurse,  patient  self-report,  and  social 
and  psychiatric  history  rating  scales  and  their  use  with 
psychiatric  patients.  However,  it  is  important  to  bear 
in  mind  that  projective  and  psychomotor  tests  and 
analyses  of  verbal  behavior,  to  cite  a few  examples, 
have  also  been  used,  and  continue  to  be  used  as  change 
measures  with  mental  patients. 

Advantages  of  Quantitative  Rating  Scales 

Lyerly  and  Abbott  (1965)  note  four  advantages  of 
quantitative  rating  scales.  The  first  is  as  an  interview 
guide.  All  of  the  significant  symptoms,  attitudes  and 
behaviors  are  listed  on  the  interview  form  assuring  a 
more  complete  coverage  of  possibly  relevant  data  re- 
garding the  patient.  Consequently,  there  is  less  likeli- 
hood that  significant  symptoms  or  behavior  will  be 
missed.  Ratings  scales  also  facilitate  communication 
among  professional  personnel.  Since  the  scales  are 
completed  in  a uniform  manner  and  the  items  generally 
describe  relatively  simple  and  discrete  bits  of  patient 
behavior,  the  possibility  of  misunderstanding  or  mis- 
interpretation among  psychiatrists  or  nurses  is  mini- 
mized. Rating  scales  also  provide  a record  of  treatment 
progress.  Effects  on  symptoms  and  other  behavior  as 
a consequence  of  various  treatment  procedures  may  be 
evaluated  and  trends  over  time  noted.  Finally,  as  the 
data  from  rating  scales  is  in  an  objective  and  numerical 
form,  it  can  be  subjected  to  statistical  manipulation 
and  used  for  research  purposes. 

Reliability 

The  reliability  of  a scale  or  test  may  be  measured  in 
a variety  of  ways,  however  the  term  generally  refers 
to  errors  of  measurement  in  a scale  or  test  as  reflected 
in  differences  in  ratings  or  test  scores  from  a hypo- 
thetical “true”  value.  The  smaller  these  differences  the 
more  reliable  the  scale. 


Indices  of  reliability.  One  very  simple  measure  of 
reliability  may  be  obtained  by  computing  the  correla- 
tion in  ratings  between  two  or  more  raters.  For  con- 
tinuous data,  such  as  ratings  on  a five-point  scale,  a 
Pearson  product-moment  correlation  may  be  com- 
puted. For  discrete  or  categorical  data,  true-false  items, 
for  example,  a contingency  coefficient  would  be 
suitable. 

Another  widely  used  method  of  assessing  reliabil- 
ity is  the  test-retest  coefficient;  i.e.,  the  correlation  be- 
tween sets  of  ratings  made  at  different  times.  This 
index  of  reliability  is  useful  for  measuring  the  stability 
of  traits  or  characteristics  over  time.  This  would  be 
a suitable  index  for  variables,  such  as  stable  personality 
traits,  that  are  not  expected  to  show  significant  changes 
as  a consequence  of  time.  However,  in  work  with 
mental  patients,  particularly  when  one  is  interested  in 
evaluating  the  effects  of  an  active  treatment  proce- 
dure, this  would  be  a poor  measure  of  reliability  as  one 
would  expect  dramatic  changes  in  the  patient’s  be- 
havior over  time. 

Estimates  of  reliability  may  also  be  made  to  reflect 
the  extent  to  which  items  in  a scale  show  internal  con- 
sistency, i.e.,  “hang  together”  with  regard  to  an  at- 
tribute or  trait  they  are  presumed  to  measure.  Split- 
half  reliability  coefficients  are  one  example  of  a 
measure  of  internal  consistency.  The  intraclass  coeffi- 
cient is  another  example  of  an  internal  consistency 
measure.  The  intraclass  coefficient  is  particularly  use- 
ful in  situations  where  there  are  two  or  more  ratings 
for  each  patient  but  they  are  not  made  by  the  same 
set  of  raters. 

Ebel’s  (1951)  analysis  of  variance  technique  for 
estimating  intraclass  coefficients  is  a useful  device  in 
large  collaborative  studies  involving  many  raters  from 
many  hospitals.  Using  this  technique,  it  is  possible  to 
compute  a measure  of  interrater  dissimilarity  in  level 
of  ratings  (called  “level  difference”)  and  a measure  of 
interrater  dissimilarity  due  to  nonsystematic  differ- 
ences (called  “random  difference”).  Level  difference, 
the  percentage  of  level  of  total  variance,  reflects  the 
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tendency  of  one  rater  to  consistently  rate  the  patient 
higher  or  lower  than  the  other  rater.  Random  differ- 
ence is  the  percentage  of  error  variance,  excluding  level 
variance,  to  total  variance. 

The  question  of  how  large  a reliability  coefficient 
should  be  to  justify  the  use  of  a scale  or  test  has  no 
simple  answer.  For  clinical  purposes  involving  deci- 
sions on  the  treatment  of  a patient,  the  clinician  wants 
quite  high  reliability,  i.e.,  in  the  80’s  or  90’s.  For  re- 
search purposes,  to  demonstrate  treatment  effects, 
lower  reliabilities  can  be  tolerated. 

How  can  one  enhance  or  increase  the  reliability  of  a 
rating  instrument  or  conversely  what  are  some  of  the 
sources  of  unreliability  in  rating  scales?  First  and  fore- 
most, the  scale  items  must  be  unambiguous,  and  insofar 
as  possible,  should  not  rely  on  clinical  judgments.  In 
other  words,  the  item  should  “mean”  the  same  thing 
to  the  different  raters.  Occasionally,  this  criterion  is 
difficult  to  achieve  and  can  only  be  achieved  after  a 
series  of  training  sessions  with  the  raters.  Secondly,  the 
items  should  describe  relatively  simple  and  discrete  bits 
of  behavior.  Thirdly,  a number  of  psychiatric  rating 
scales  such  as  the  Brief  Psychiatric  Rating  Scale 
(BPRS;  Overall  and  Gorham,  1960)  and  the  Inpa- 
tient Multidimensional  Rating  Scale  (IMPS;  Lorr, 
Klett,  McNair,  & Lasky;  1963)  are  completed  follow- 
ing a standard  psychiatric  interview  with  the  patient. 
Consequently,  the  ratings  are  based  on  recall.  If  the 
rater  is  not  too  familiar  with  the  rating  instruments  it 
is  also  possible  he  may  have  failed  to  question  the 
patient  in  certain  areas  and  so  is  unable  to  answer 
some  of  the  questions.  A better  approach  would  be  to 
fill  out  the  form  while  conducting  the  usual,  relatively 
unstructured  interview  and  to  then  ask  the  patient 
more  structured  questions  to  cover  items  missed  during 
the  initial  phase  of  the  interview.  Finally,  the  authors 
of  some  psychiatric  rating  scales  insist  on  independent 
ratings  from  two  raters  to  enable  the  investigator  to 
estimate  the  reliability  of  the  two  raters.  Consequently, 
the  raters  are  specifically  instructed  not  to  discuss  the 
interview  with  the  patient  prior  to  completing  their 
ratings.  However,  it  is  conceivable  that  more  valid 
estimates  of  the  patient’s  psychopathology  might  be 
obtained  if  the  raters  were  encouraged  to  discuss  their 
impressions  of  the  patient  and  they  then  completed  a 
single  form  where  the  ratings  represented  their  con- 
sensus of  the  patient’s  interview  behavior. 

Validity 

Simply  stated,  validity  is  a measure  of  the  extent  to 
which  the  test  or  scale  actually  measures  what  it  pur- 
ports or  proposes  to  measure. 


Predictive  validity.  This  type  of  validity  is  generally 
of  greatest  concern  and  import  to  the  typical  user  of 
a rating  scale.  If  an  investigator  is  interested  in  a par- 
ticular outcome  or  criterion,  he  wants  the  scale  to  accu- 
rately measure  this  outcome.  The  scale  should  be  suffi- 
ciently sensitive,  for  example,  to  reflect  the  differential 
effects  of  two  or  more  treatments  on  outcome  after 
three,  five,  seven,  etc.,  weeks  of  treatment. 

Inexperienced  investigators  often  err  in  selecting  a 
scale  or  test  that  has  high  predictive  validity  for  the 
uses  originally  intended  but  has  low  predictive  validity 
for  the  particular  study  of  the  investigator.  For  ex- 
ample, an  investigator  may  be  interested  in  measuring 
psychopathology  in  depressed  patients  but  selects  a 
scale  originally  designed  for  use  with  schizophrenic 
patients  and  that  does  not  include  many  of  the  symp- 
toms specific  to  depression. 

Concurrent  validity.  This  type  of  validity  is  con- 
cerned with  the  comparability  of  two  sets  of  measures 
determined  at  about  the  same  time.  The  extent  to 
which  scale  scores  can  differentiate  currently  diagnosed 
groups  of  psychiatric  patients  is  an  example  of  this 
type  of  validity. 

Content  validity.  Here  the  concern  is  with  the  repre- 
sentativeness of  a selected  sample  of  items  randomly 
drawn  from  a parent  pool  to  that  parent  pool. 

Construct  validity.  The  test  or  scale  is  designed  to 
measure  a hypothesis  that  is  itself  not  directly  testa- 
ble. However,  if  the  results  with  the  test  are  consistent 
with  the  hypothesized  results,  this  is  considered  an 
index  of  construct  validity. 

Norms 

Unfortunately,  few  authors  of  rating  scales  for  men- 
tal patients  have  reported  norms  for  their  scales.  Fur- 
ther, the  norms  that  have  been  reported  are  difficult  to 
evaluate  because  they  are  generally  based  on  small 
sample  sizes  and  from  one  or  a few  hospitals.  To  date, 
the  most  dependable  norms  have  been  established  by 
the  Veterans’  Administration  based  on  the  results  of 
a number  of  large  collaborative  studies.  However,  re- 
search involving  the  use  of  rating  scales  can  be  carried 
out  successfully  without  norms,  provided  the  tests 
or  measures  have  sufficient  reliability  or  validity. 

Rating  Scales 

Rating  scales  for  measuring  change  in  mental  pa- 
tients can  be  categorized  in  a variety  of  ways.  These 
include  the  person  doing  the  ratings,  e.g.,  psychiatrist, 
nurse  or  patient,  the  patient  population  to  be  sampled, 
e.g.,  depressions  or  schizophrenias,  the  content  of  the 
scale,  e.g.,  symptom  rating  scales  or  scales  to  evaluate 
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the  patient’s  social  and  psychiatric  history,  and  the 
format,  e.g.,  ratings  based  on  a structured  or  unstruc- 
tured interview.  For  heuristic  purposes,  the  selected  rat- 
ing scales  to  be  presented  in  this  paper  were  grouped 
on  the  basis  of  the  person  doing  the  ratings. 

Psychiatric  rating  scales.  In  general,  scales  in  this 
category  are  intended  for  use  by  either  psychiatrists 
or  psychologists.  Some  were  designed  with  specific 
patient  populations  in  mind  and  others  may  be  used 
with  a wider  variety  of  psychiatric  patients.  The  Hamil- 
ton Scale  (Hamilton,  1960)  and  Beck  Depression  In- 
ventory (Beck,  1961)  are  examples  of  instruments  de- 
signed for  a specific  patient  population,  namely  de- 
pressions. The  Wittenborn  Psychiatric  Rating  Scale 
(Wittenborn,  1955)  and  the  latest  version  of  the  In- 
patient Multidimensional  Psychiatric  Scale  (IMPS) 
could  conceivably  be  used  with  depressives  as  well  as 
schizophrenics.  However,  there  is  no  truly  omnibus 
rating  scale  that  is  entirely  appropriate  for  all  patient 
populations.  The  Wittenborn  Scale,  for  example,  sam- 
ples some  ward  behaviors  and  therefore  is  not  suitable 
for  outpatients.  The  IMPS  is  heavily  loaded  with  severe 
forms  of  psychopathology  and  would  not  be  too  appro- 
priate for  rating  neurotic  or  less  severely  ill  patients. 
Some  of  these  instruments  are  designed  to  provide  a 
total  morbidity  score  (the  Hamilton  and  Beck  Scales) 
whereas  others  provide  factor  scores  in  certain  symp- 
tom areas.  The  IMPS,  for  example,  provides  factor 
scores  characterizing  10  basic  dimensions  of  psycho- 
pathology. Other  rating  scales  such  as  the  Wittenborn 
have  also  been  factor  analyzed.  Most  of  these  rating 
scales  are  completed  following  the  usual  mental  status 
interview.  Spitzer,  Fleiss,  Burdock  and  Hardesty 
(1964)  developed  a structured  interview  in  which  the 
interviewer  asks  the  patient  specific  questions  and  his 
replies  are  coded  as  the  interview  progresses.  In  keep- 
ing with  the  times,  a number  of  these  scales  have  also 
undergone  format  changes  to  permit  mechanical  scor- 
ing and  automatic  transfer  of  the  scores  to  IBM  cards. 

Nurse  rating  scales.  Three  scales  were  designed  spe- 
cifically to  measure  the  patient’s  ward  behavior.  These 
are  the  Burdock  Ward  Behavior  Scale  (BWBS;  Bur- 
dock, Hakerem,  Hardesty  and  Zubin,  1960)  intended 
originally  for  use  with  schizophrenic  patients  but  re- 
cently adapted  by  NIMH  for  use  with  depressions,  the 
Psychotic  Reaction  Profile  (PRP;  Lorr,  O’Connor  and 
Stafford,  1961)  and  the  NOSIE  (Honigfeld,  Gillis  and 
Klett,  1966).  The  PRP  and  NOSIe"  were  designed 
mainly  for  rating  schizophrenic  patients.  Factor  scores 
are  available  for  the  BWBS  and  the  PRP.  The  NOSIE 
consists  of  only  30  items  and  has  not  been  factor 
analyzed. 


In  contrast  to  the  psychiatric  rating  scales,  which 
are  generally  rated  following  an  interview  with  the 
patient,  the  ward  behavior  scales  encompass  a longer 
period  of  observation,  generally  one  week.  Conse- 
quently, it  is  important  that  these  be  completed  by  a 
nurse  who  is  very  familiar  with  the  patient,  preferably 
the  patient’s  ward  nurse  rather  than  a nurse  assigned 
to  the  research  project. 

Patient  self-report  forms.  It  is  possible  to  devise  a 
simple  symptom  checklist,  suitable  for  use  by  either 
patients  or  psychiatrists.  An  example  of  such  an  instru- 
ment is  the  Symptom  Checklist  (Parloff,  Kelman  and 
Frank,  1954).  However,  in  general,  psychiatric  rating 
scales  contain  sophisticated  psychiatric  terminology 
that  makes  them  unsuitable  for  use  with  patients.  A 
very  popular  instrument  that  is  usable  with  patients 
and  can  be  rated  by  psychiatrists  or  nurses,  is  the  ad- 
jective checklist.  The  patient  is  asked  to  rate  adjectives 
such  as  sad,  nervous,  and  tense  on  a frequency  of  oc- 
currence or  intensity  scale.  Many  of  these  checklists 
have  been  factor  analyzed  and  factors  characterizing 
anxiety,  depression,  hostility,  etc.,  identified.  The  Clyde 
Mood  Scale  (Clyde,  1963)  is  a representative  example. 
A number  of  adjective  checklists  also  have  multiple 
forms  which  make  them  preferable  for  use  in  repeated 
measurement  of  designs  where  the  patient  may  be  re- 
quired to  fill  out  the  form  on  a daily  basis  (Zuckerman, 
1960) . A major  problem  with  all  patient  report  inven- 
tories is  the  tendency  for  the  patient  to  rate  himself 
either  high  or  low  in  all  areas  of  psychopathology.  This 
tendency  has  been  called  the  “sick  set.”  Consequently, 
when  the  instrument  is  factor  analyzed  you  tend  to  get 
one  very  large  general  factor  reflecting  this  set.  This 
set  may  be  lessened  by  including  items  that  reflect 
mental  health  and  well-being  as  well  as  items  sampling 
psychopathological  behaviors. 

Social  worker  rating  scales.  Generally,  these  scales 
are  not  designed  specifically  for  use  by  a social  worker 
but  are  intended  to  evaluate  the  patient’s  premorbid 
social  and  psychiatric  history.  However,  traditionally 
they  have  been  completed  by  a study  social  worker  fol- 
lowing an  interview  with  a family  informant.  Obvi- 
ously, if  this  data  is  to  be  valid  it  is  important  that  the 
family  informant  be  a close  family  member,  and  also 
be  a reliable  informant.  Unfortunately,  few  family 
informants  meet  these  criteria.  For  example,  many 
wives  have  little  knowledge  of  their  husband’s  work 
adjustment  and  ability  to  get  along  with  coworkers  and 
supervisors. 

Although  investigators  are  generally  aware  of  the 
need  to  have  training  sessions  with  psychiatrists  and 
psychologists  to  assure  reasonable  reliability  on  the 
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psychiatric  rating  forms,  they  tend  to  overlook  the 
need  to  hold  training  sessions  with  the  study  social 
workers.  Consequently,  these  individuals  often  rate 
items  such  as  the  number  of  prior  psychiatric  episodes 
from  quite  different  frames  of  reference.  A recent  in- 
novation with  social  and  psychiatric  history  records  is 
the  attempt  to  adapt  these  forms  to  facilitate  transfer 
of  the  ratings  to  IBM  cards. 


One  instrument,  the  Katz  Adjustment  Scale  (Katz 
and  Lyerly,  1963)  was  specifically  designed  for  com- 
pletion by  a family  informant.  The  informant  rates  a 
series  of  items  indicative  of  mild,  moderate,  and  severe 
psychopathology  as  typical  or  not  typical  of  the  pa- 
tient “Now”  and  “All  his  life.”  This  instrument  has 
also  been  factor  analyzed  and  some  normative  data 
are  available. 
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PROGRAM  EVALUATION:  A PROGRAM  DIRECTOR’S  VIEWPOINT 

Eugene  A.  Hargrove,  M.D. 

North  Carolina  Department  of  Mental  Health 


Before  attempting  to  fix  responsibility  for  program 
evaluation,  I think  it  might  be  appropriate  if  I try  to 
define  for  you  what  I conceive  program  evaluation  to 
be  and  what  I think  it  is  not. 

I think  of  program  evaluation  as  involving  a con- 
tinuing correction  of  one’s  program  based  upon  analysis 
of  oneself  in  relation  to  one’s  goals.  I want  to  emphasize 
that  to  me  program  evaluation  is  not  just  running 
experimental  studies.  It  is  a process  through  which  we 
make  decisions  concerning  the  steps  which  we  take  in 
reaching  our  goals. 

At  least  three  essential  elements  are  required  in  order 
to  carry  on  program  evaluation.  The  first  of  these  is 
a statement  of  goals.  It  should  be  noted  that  no  pro- 
gram can  be  called  purposeful  without  a set  of  goals, 
and  if  we  think  of  a mental  health  organization  as  a 
purposeful  program  then  it  should  be  able  to  state  its 
goals.  We  should  recognize  that  a statement  of  goals 
is  a difficult  and  time-consuming  task,  but  one  essential 
for  any  future  evaluation  of  the  program  and  a deter- 
mination of  its  ability  to  function  optimally.  In  mental 
health  these  goals  must  be  as  operational  as  possible  to 
enable  continuous  evaluation  and  a determination  of 
the  degree  of  goal  achievement  or  nonachievement 
to  occur.  It  is  one  thing  to  state  general  goals  such  as 
“to  restore  the  mentally  ill  to  self-dependence  and  op- 
timum well  being”  or  “to  prevent  mental  illness”  or 
“to  maintain  a responsiveness  to  public  need.”  Such 
goals  are  more  directional  than  operational;  they  are 
not  stated  in  a framework  conductive  to  evaluation, 
short  of  subjective  judgment. 

My  position  is  that  these  goals  must  by  their  very 
necessity  become  as  quantitative  as  possible;  that  is, 
they  must  be  quantitative  standards  which  are  indices 
of  the  more  general  qualitative  goals.  For  example,  it 
is  one  thing  to  say,  “our  goal  is  to  prevent  alcoholism” ; 
it  is  quite  another  thing  to  say  “I  will  reduce  the  inci- 
dence of  alcoholism  in  a year  by  25  percent  for  the  en- 
tire population.”  The  latter  is  more  operational  in 
evaluation.  Not  only  should  our  goals  be  as  operational 


as  possible ; they  must  not  be  confused  with  the  meth- 
ods devised  to  achieve  these  goals. 

Goals  may  be  stated  at  a variety  of  levels  which 
would  include  overall  organizational  goals,  goals  of 
various  subprograms,  and  even  individual  goals  of 
the  various  participants.  We  should  note  again  that 
general  qualitative  goals  are  desirable  to  give  direction 
and  scope  but  such  goals  become  much  more  meaning- 
ful if  quantitative  indices  can  be  developed  for  them. 

While  the  statement  and  restatement  of  goals  is  an 
ongoing  process,  it  is  my  recommendation  that  an  ini- 
tial and  tentative  set  of  goals  be  established  by  each 
subprogram  and  for  the  total  program.  Such  goal  state- 
ments can  result  only  from  directed,  coordinated  ef- 
forts involving  participants  according  to  various  levels 
of  responsibility.  Both  the  definition  of  the  program 
and  the  statement  of  goals  are  essential  to  effective 
evaluation. 

We  have  said  then  that  the  first  essential  element  for 
program  evaluation  is  a clear  statement  of  where  you 
want  to  be,  or  your  goals. 

The  second  essential  is  a means  to  determine  where 
you  are  now.  This  requires  feedback,  which  represents 
the  analysis  of  information  about  a system’s  behavior 
as  a guide  for  future  behavior.  The  existence  of  feed- 
back presupposes  that  there  exists  first  an  information 
gathering  capability  of  the  program  and  secondly,  an 
ability  to  analyze  information.  The  development  of 
feedback  capabilities  in  a mental  health  system  should 
be  built  on  both  abilities,  for  comprehensive  informa- 
tion is  essential  to  any  purposeful  program  which  seeks 
optimal  or  “best  possible”  behavior. 

Information  analysis  incorporates  the  logic  of  any 
feedback  capability.  It  makes  all  unstructured  messages 
meaningful  and  useful  information.  It  is  important  to 
recognize  both  the  ability  to  gather  and  analyze  infor- 
mation. 

The  third  element  needed  in  program  evaluation  is 
logical  decisions  as  to  how  to  reach  the  goals  that  you 
have  set.  Decisions  about  goals,  indices  of  goals,  correc- 
tions which  are  necessary  to  make  our  activities  more 
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goal  directed — are  the  fabric  of  evaluation,  the  raw 
material  on  which  all  other  activities  identified  as  eval- 
uation are  based.  Because  this  is  so,  it  is  well  to  empha- 
size that  evaluation  as  a decision  task  lies  squarely  in 
the  administrators  area  of  responsibility,  not  some- 
thing which  can  be  scuttled  off  to  the  side  by  giving 
“evaluation”  responsibilities  to  a research  or  another 
staff  person. 

Essential  to  the  decision  making  process  is  a network 
to  gather  and  disseminate  the  messages  needed  by  the 
program  to  properly  function.  An  important  part  of 
this  component  is  the  determination  of  the  information 
needed  by  each  decision  point,  with  a concerted  effort 
to  make  such  information  as  relevant  and  as  current  as 
possible.  No  decision  maker  can  operate  effectively  with 
information  that  is  old  or  lacks  currency.  He  can  hardly 
make  decisions  about  the  kinds  of  activities  needed  in  a 
particular  community  to  meet  the  mental  health  needs 
unless  as  much  information  as  possible  is  available 
about  the  problems  that  exist  in  the  community,  about 
available  resources  within  the  community  to  meet  these 
problems,  and  about  the  general  social-cultural  struc- 
ture in  which  any  formal  effort  at  meeting  these  needs 
must  be  incorporated. 

A program  of  message  retrieval  and  dissemination  is 
largely  dependent  upon  the  types  of  program  defini- 
tions and  goal  statements  which  have  preceded  any 
serious  involvement  in  a total  message  network.  A large 
part  of  this  program  should  be  directed  at  seeking  in- 
novative means  to  carry  out  this  function.  The  high- 
speed computer  has  made  it  possible  for  a decision- 
maker to  have  available  comprehensive  and  detailed 
information  about  the  decision  situation  and  obtain 
such  information  as  rapidly  as  possible.  But  simply  to 
put  on  a computer  those  things  we  are  currently  doing 
in  the  way  of  statistics  and  data  analysis  is  to  assume 
that  such  data  are  the  most  meaningful  that  can  be 
produced.  Such  efforts  often  tend  to  complicate  our 
own  ignorance  about  what  it  is  that  we  do  and  do  not 
know  about  ourselves;  that  is,  do  what  we  do  incor- 
rectly much  faster.  It  is  my  belief  that  no  compre- 
hensive efforts  at  use  of  high-speed  computers  should 
be  made  until  serious  inroads  have  been  made — in  de- 
fining our  program  and  stating  its  goals.  It  is  clear, 
however,  that  program  definitions  and  goal  statements 
may  be  facilitated  by  a more  meaningful  presentation 
of  available  statistics,  but  such  effort  should  be  looked 
upon  as  simply  stop-gap. 

I think  we  all  recognize  that  evaluation  for  mental 
health  or  other  related  service  organizations  involves 
unique  problems  which  are  not  characteristic  of,  say,  a 
free  enterprise  industrial  concern.  In  evaluating  its 
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goal  achievement,  industry  turns  to  its  profit-loss  state- 
ment. If  that  statement  is  running  in  the  black  and  if 
the  stockholders  are  making  a fair  return  on  their 
investment,  the  industry  can  be  regarded  as  achieving 
its  goals.  We  in  mental  health,  on  the  other  hand, 
have  not  yet  evolved  a strategy  for  defining  our  goals. 
Indeed  the  very  size  and  complexity  of  our  operations 
discourage  us  from  searching  for  an  operational  defi- 
nition of  our  program,  a definition  that  we  can 
evaluate. 

I think  the  time  is  coming,  however,  and  may  even 
be  upon  us,  when  we  are  going  to  be  expected  and 
required  to  account  to  the  people  through  their  gov- 
ernmental leaders  in  a much  more  precise  way  for  the 
efficiency  or  the  inefficiency,  with  which  we  are  spend- 
ing our  tax  funds  for  mental  health  programs.  In  order 
to  do  this  effectively,  we  are  going  to  have  to  press  on 
with  the  difficult,  complex,  and  often  dirty  business  of 
making  evaluation  an  integral  part  of  all  of  our 
programs. 

Inasmuch  as  we  have  very  little  precedent  for  the 
process  of  evaluation  in  this  field  or  any  other  service 
field  for  that  matter,  we  must  be  prepared  not  only  to 
engage  in  the  painful  process  of  setting  goals  but  we 
must  be  willing  to  change  as  a result  of  our  own  evalu- 
ative findings.  This  is  perhaps  the  most  difficult  thing 
for  a service  agency  to  do,  since  we  have  a very  long 
tradition  of  expansion  but  not  much  tradition  of  direct 
change,  including  amputation  if  necessary. 

With  this  as  background,  I am  ready  to  turn  now 
to  a discussion  of  whose  responsibility  program  evalu- 
ation is. 

It  seems  obvious  to  me  that  there  are  several  differ- 
ent levels  of  responsibility  in  evaluation  and  that  a 
number  of  different  disciplines  will  necessarily  be 
involved.  Speaking  from  the  standpoint  of  a State  com- 
missioner of  mental  health,  however,  I have  a respon- 
sibility for  creating  the  kind  of  atmosphere  wherein 
the  process  of  program  evaluation  can  begin  to  take 
place  at  every  level  up  the  line  to  the  local  level  of 
operation.  I think  it  is  essential  that  goal  definitions 
begin  first  at  the  top  level  in  order  that  programs  at 
other  levels  can  set  their  goals  which  are  compatible 
with  those  of  the  goals  and  objectives  of  the  statewide 
program. 

I also  believe  it  wise  to  have  a State-level  profes- 
sional who  can  coordinate  the  planning  and  evaluation 
process  and  provide  direct  consultation  to  the  State 
program  administrator.  It  would  seem  important  to 
keep  this  person  free  of  direct  involvement  in  evalu- 
ation, although  he  might  become  involved  in  some 
pilot  efforts,  in  order  to  get  the  process  started.  I would 
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compare  his  function  more  to  that  of  the  skilled 
mechanic.  Like  the  automobile  mechanic,  the  State- 
level  evaluation  professional  keeps  the  machine  in  re- 
pair and  running  at  top  efficiency  but  he  does  not 
actually  drive  the  car.  He  acts  as  the  catalyst  who  pre- 
cipitates the  evaluation  process,  and  who  remains  on 
call  for  maintenance  and  help  if  trouble  develops. 

Moving  from  the  State-level  organizational  struc- 
ture, I can  see  the  need  for  a group  at  the  next  level 
whose  responsibility  it  would  be  to  get  down  to  the 
nitty-gritty  of  program  evaluation.  These  people  would 
be  assigned  areas  of  responsibility  either  on  a regional 
basis  or  according  to  broad  program  areas — children, 
adolescents,  alcoholics,  adults,  geriatrics,  etc.  They 
would  relate  to  the  State-level  evaluation  consultant, 
and  at  the  unit  or  patient  level  to  that  program  admin- 
istrator (i.e.  the  hospital  unit  director,  the  community 
clinic  director,  etc.) . It  is  my  belief  that  most  of  these 
program  administrators  are  so  burdened  with  the  day- 
to-day  responsibilities  of  program  administration  that 
they  do  not  have  time  to  initiate  the  evaluation 
process.  So  without  trained  people  having  clear  respon- 
sibility to  involve  themselves  with  program  administra- 
tors in  the  tough  work  of  goal  definition,  development 
of  informational  channels,  and  decisionmaking,  I am 
afraid  that  the  evaluation  process  will  never  become 
an  integral  part  of  our  total  mental  health  system. 

I would  not  limit  the  evaluation  professionals  to  a 
task  orientation  only.  I suggest  that  they  be  seen  as 
information  seekers,  organizers,  and  disseminators  as 
well.  This  would  include  efforts  at  educating  all  pro- 
fessionals in  the  notions  of  decisionmaking;  the  con- 
ceptual schemes  for  organizing  data,  such  as  statistics; 
the  philosophical  issues  surrounding  our  goal  state- 
ments; as  well  as  organizing  and  maintaining  the 
sources  of  information  on  which  evaluation  decisions 
can  be  made. 

This  is  not  to  say  that  the  subprogram  administrator 
does  not  have  some  very  definite  responsibilities  in  this 
area.  First,  he  must  be  sold  on  evaluation  as  a process 
and  must  be  willing  to  give  this  his  wholehearted 
administrative  support,  communicating  this  to  his  staff. 
He  must  be  willing  to  devote  time  and  real  soul- 
searching  effort  to  defining  as  specifically,  as  con- 


cretely and  as  quantifiably  as  possible  what  his  program 
is  trying  to  do  for  the  patient;  in  other  words,  what  his 
goals  are.  The  administrator  must  be  willing  to  aid  in 
the  search  for  criteria  that  will  give  leads  on  whether 
or  not  the  goals  are  being  achieved.  And  finally,  he 
must  be  flexible  enough  and  willing  to  modify  his 
program  in  light  of  the  findings  of  the  evaluation 
process. 

The  third  and  equally  important  member  of  this 
evaluation  triad  is  the  statistical  analyst.  I would  see 
the  statistician,  in  the  same  way  as  the  evaluator,  relat- 
ing either  regionally  or  to  the  broad  program  areas 
(children,  adolescents,  adults,  etc.).  In  some  cases,  the 
roles  of  the  evaluator  and  the  statistical  person  might 
blur,  where  you  may  find,  for  example,  a person  skilled 
in  evaluation  who  also  is  a good  statistical  analyst. 
One  significant  role  of  the  statistician  is  that  of  data 
analysis,  directed  especially  toward  measuring  criteria 
which  will  reflect  program  effects. 

In  closing,  let  me  stress  the  need  for  an  organized 
and  systematic  effort  to  implement  evaluative  pro- 
cesses. Since  this  is  rather  new  to  us  in  mental  health,  it 
should  be  phased  in  order  to  work  progressively  and 
meaningfully  to  larger  and  larger  spheres  of  applica- 
tion. Certainly,  its  introduction  should  be  accompanied 
by  orientation.  Administrators  need  to  understand  the 
logic  on  which  evaluation  is  based  rather  than  some- 
thing “handed  down  from  above.”  As  efforts  to  form- 
ally evaluate,  extend  further  and  further  into  the 
total  organization,  they  should  always  be  preceded 
by  education. 

The  structure  which  I have  outlined  is  not  neces- 
sarily the  only  one  wherein  program  evaluation  can 
take  place.  I am  sure  that  others  of  you  will  approach 
this  differently,  depending  upon  the  uniqueness  and 
variability  of  your  own  programs.  I do  think,  how- 
ever, that  in  whatever  ways  we  may  implement  it, 
program  evaluation  must  begin  to  permeate  the  en- 
tire mental  health  system.  Only  through  this  process 
can  we  begin  to  respond  to  the  mandate  which  we 
are  being  given  to  optimize  our  operations  in  behalf 
of  the  mentally  ill,  the  alcoholic  and  the  mentally 
retarded  and  to  obtain  the  desired  sensitivity  to  the 
mental  health  needs  of  our  communities. 
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The  role  of  the  statistician  in  program  planning 
and  evaluation  vis-a-vis  the  administrator  and  the 
planner/evaluator  may  be  viewed  in  terms  of  effective- 
ness as  a team  member.  However,  we  should  recognize 
that  inappropriate  participation  in  pseudo  program 
planning  and  program  evaluation  occurs.  A few  of 
those  who  participate  thusly  call  themselves  researchers 
or  statisticians;  others  call  themselves  planners;  others, 
evaluators;  and  still  others,  administrators.  Planning 
and  evaluation  are  trumpeted  about  noisily  by  many 
who  see  (and  observe)  little,  write  much,  and  blat 
more.  The  overlooked  ingredient  is  that  effective  par- 
ticipation is  not  always  equatable  to  mere  observable 
participation. 

Effective  participation  of  statisticians  in  program 
planning  and  evaluation  requires  that  we  be  aware 
that  effectiveness  may  well  be  dependent  upon  the 
bases  of  power  in  decisionmaking.  Interrelated,  but 
with  more  specificity  to  statisticians,  are  the  following: 
the  effects  of  motivation  and  competence;  working 
knowledge  of  both  technical  and  general  subject  areas ; 
and  ability  to  provide  useful  consultation. 

When  decisionmaking  involves  more  than  one  per- 
son, as  is  true  in  both  program  planning  and  program 
evaluation,  participants  should  be  aware  of  what  con- 
stitutes the  bases  for  the  making  of  decisions.  More  than 
20  years  ago  I read  some  of  Max  Weber’s  writings 
which  developed  the  idea  of  a trichotomy : traditional, 
charismatic,  and  bureaucratic  authority.  Subsequently, 
other  writers  have  enlarged  upon  the  concepts  and 
added  other  dimensions.  Much  may  be  found  in  works 
by  such  authors  as  Chester  Barnard,  Peter  M.  Blau, 
Theodore  Caplow,  Peter  F.  Drucker,  Amitai  Etzioni, 
John  W.  Gardner,  Jerald  Hage,  and  C.  Northcote 
Parkinson.  Recently,  A.  C.  Filley  and  A.  J.  Grimes(7) 
identified  a number  of  bases  of  power  by  studying  ques- 
tionnaire and  interview  responses  from  employees  in  a 
nonidentified,  nonprofit  professional  organization.  The 
study  elicited  responses  in  terms  of  perceived,  preferred, 
and  utilized  organizational  structure.  Hypothetical  case 


situations  were  described  and  followed  by  three  ques- 
tions : 

1.  To  whom  should  you  go  for  permission,  consulta- 
tation,  decision,  etc.  (depending  upon  the  situa- 
tion) and  why?  This  related  to  the  perceived 
organizational  structure. 

2.  To  whom  would  you  like  to  go  for  permission,  etc., 
and  why?  Preferred  organizational  structure. 

3.  To  whom  would  you  in  fact  go  for  permission,  etc., 
and  why?  Utilized  organizational  structure. 

Filley  and  Grimes  delineated  12  bases  of  power;  inter- 
estingly, none  related  to  charisma:  (2) 

1.  Formal  authority.  There  is  actual  or  perceived 
legitimacy. 

2.  Responsibility  and  function.  Responsibility  for  an 
activity  may  or  may  not  be  associated  with  a job 
title. 

3.  Manipulation.  The  decisionmaker  will  take  some 
type  of  action  seen  as  desirable  by  the  requester. 

4.  Default  or  avoidance.  Another  appropriate  or  de- 
sired source  of  decisionmaking  is  not  available. 

5.  Autonomy.  The  respondent  does  not  feel  he  has  to 
see  anyone  else. 

6.  Expertise.  The  decisionmaker  is  felt  to  be  closer  to 
or  has  better  knowledge  of  a situation,  or  he  has 
special  ability. 

7.  Control  of  resources.  The  decisionmaker  controls 
money,  general  information,  procedures,  etc. 

8.  Bureaucratic  rules.  Policies,  regulations,  channels, 
etc.,  are  already  established. 

9.  Traditional  rules.  Reliance  is  placed  upon  custom 
or  seniority. 

10.  Collegial.  Decision  is  made  through  consultation 
with  a group  of  peers  or  through  some  kind  of 
group  action  or  participation. 

11.  Friendship.  The  respondent  has  personal  relation- 
ships or  feelings  of  good  will  toward  the  decision- 
maker. 

12.  Equity.  The  decisionmaker  is  known  to  be  fair  in 
dealing  with  problems. 
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Classifications  appropriate  to  statisticians  probably 
would  include  formal  authority,  responsibility  and 
function,  expertise,  control  of  resources,  collegial, 
friendship,  and  equity.  Of  that  group,  statisticians  may 
be  best  able  to  increase  the  bases  of  power  relating  to 
expertise,  collegial,  friendship,  and  equity.  (Of  course, 
the  statistician  should  bear  in  mind  the  fact  that  au- 
thority is  not  the  same  as  leadership.  Leadership  is 
more  difficult  to  exert  than  authority.  No  one  has  to 
follow  a leader;  he  may  or  may  not  have  legitimacy; 
a leader  can  only  ask  while  the  person  with  authority 
can  require. ) The  statistician  becomes  increasingly  ef- 
fective as  he  acquires  reputation  for  having  expert 
knowledge  and  being  equitable.  Through  the  collegial 
process  and  through  demonstrable  actions,  he  can 
build  the  basis  for  improved  personal  relationships 
with  the  administrator  and  the  program  planner /eval- 
uator. He  can  also  simultaneously  pave  the  way  toward 
increasing  power  bases  relating  to  responsibility  and 
function.  Thus,  he  can  take  advantage  of  his  knowl- 
edge of  the  bases  of  power  in  decisionmaking.  The  end 
result  will  be  that  he  will  play  a more  effective  staff 
role  in  program  planning  and  program  evaluation. 

More  specific  to  the  statistician  and  related  closely 
to  the  development  of  expertise  are  considerations  of 
the  effects  of  motivation  and  competence;  technical 
knowledge  relating  to  statistics  and  research;  general 
subject  knowledge,  and  the  ability  to  provide  consul- 
tative services. 

Motivation  has  been  defined  by  George  Huber  (5) 
of  the  University  of  Wisconsin  College  of  Engineering 
as  being  “The  inclination  to  satisfy  a need  . . . 
(and)  ...  in  part  a function  of  the  professional’s 
previous  experiences  which  have  shaped  the  hierarchy 
and  intensity  of  his  needs  and  the  perception  he  has 
of  his  ability  to  satisfy  these  needs.”  Huber  posits  that 
the  level  of  general  motivation  can  be  increased.  He 
suggests  that  this  can  be  accomplished  by  meeting  or 
exceeding  other  than  trivial  goals  and  by  minimizing 
the  cost  of  failure.  The  statistician  may  raise  his  own 
general  motivation,  then,  by  setting  and  meeting  goals 
which  are  realistically  commensurate  with  his  current 
abilities  and  functioning.  He  can  force  his  level  of 
aspirations  in  an  upward  direction  without  outside 
intervention.  (4)  Hopefully,  as  successes  are  met, 
upper  administrative  levels  will  become  aware  of  such 
successes  and  help  to  reinforce  the  positive  attitudes 
which  will  accrue.  Of  interest  to  us  is  Huber’s  proposi- 
tion that  “people  increase  the  range  and  strength  of 
their  coping  efforts  if  their  role  requirements  are 
great.” 


Competence  relates  to  the  ability  to  carry  out  assign- 
ments satisfactorily.  For  statisticians  that  means  prob- 
lem solving  in  statistics  and  research  and  effective 
exercise  of  interpersonal  relationships.  Problem  solving 
requires  good  utilization  of  both  technical  and  general 
knowledge.  We  should  apply  an  orderly  research  ap- 
proach (viz.,  state  the  problem  in  researchable  terms; 
make  preliminary  observations  and  find  out  what  others 
have  done  about  the  problem;  develop  hypotheses  or 
ways  of  meeting  the  problem;  decide  on  methods, 
timing,  costs,  etc.;  and  then  carry  out  the  plans).  All 
that  is  easier  said  than  done  and  was  so  recognized 
in  a paper  (5)  presented  during  the  October  1967 
Southern  Regional  Conference  on  Mental  Health 
Statistics.  However,  application  of  an  orderly  research 
approach  coupled  with  mastery  of  technical  knowledge 
is  not  enough  to  meet  needs  demanded  by  evaluative 
research.  Basic  conceptualizations  must  not  be  faulty 
or  too  narrow;  a comprehensive  study  must  not  be 
designed  when  only  a limited  amount  of  data  is  needed 
to  answer  specific  administrative  questions. 

The  statistician  must  exercise  both  judgment  and 
tact  when  working  with  others  in  eliciting  what  is 
really  needed  when  program  evaluation  studies  are 
being  planned.  Subsequently,  he  must  be  able  to  exer- 
cise a considerable  amount  of  flexibility  in  that  he  will 
probably  lose  “control  over  the  research  situation  . . . 
(in  that)  . . . someone  else,  usually  a program  ad- 
ministrator with  a strong  vested  interest,  has  already 
defined  the  objectives  of  the  program  to  be  evaluated. 
To  force  him  to  question  the  underlying  assumptions 
of  these  objectives  is  both  difficult  and  painful.”  ( 6 ) 
Another  consideration  was  pointed  out  by  Levy,  Her- 
zog, and  Slotkin.  (7)  “One  of  the  first  requirements  of 
evaluative  research  in  a State  agency  context  has  been 
the  need  to  develop  a conceptual  model  of  the  evalua- 
tion process  so  as  to  avoid  being  overwhelmed  by  the 
enormous  variety  of  data  that  affect  our  activities.  We 
have  found  that  conceptualizing  evaluation  as  one  ele- 
ment in  a systems  process  model  is  helpful  in  providing 
an  overview.”  Thus,  in  the  current  situation,  compe- 
tence in  evaluation  research  relates  to  knowledge  and 
application  of  systems  theory. 

It  appears  that  the  longer  we  are  in  this  business 
of  being  mental  health  statisticians,  the  more  we  have 
to  know.  Our  work  now  requires  that  we  have  knowl- 
edge in  such  areas  as  statistics,  probability  theory,  sys- 
tems analysis,  and  computer  usage.  In  Wisconsin  we 
are  being  swept  along  by  the  tides  of  systems  analysis 
and  computer  utilization;  we  are  being  forced  to  add 
to  our  technical  knowledge;  we  are  learning.  Part  of 
what  we  are  learning  was  succinctly  summarized  in  a 
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recent  issue  of  Business  Automation.  ( 8 ) Some  of  the 
basic  problems  in  making  a total  management  infor- 
mation system  work  were  categorized  as  follows : 

“1.  Overabundance  of  irrelevant  information — tend- 
ency to  fail  to  isolate  the  key  factors. 

“2.  Insufficient  planning  or  development — -tendency 
to  push  for  results  too  quickly. 

“3.  Oversophistication — lack  of  a complete  under- 
standing of  the  system  by  those  who  must  make 
it  work. 

“4.  Overconfidence — temptation  to  visualize  the  MIS 
as  ...  an  answer  for  all  problems.” 

Also  pointed  out  was  the  assertion  that  inadequate 
involvement  of  top  management  in  all  phases  was 
“probably  the  one  single  factor  that  has  brought  fail- 
ure or  disappointment  in  many  attempts  . . . .”  Thus, 
effectiveness  requires  more  than  utilization  of  techni- 
cal knowledge ; it  requires  knowledge  of  group  dynam- 
ics and  of  general  information  in  the  field  of  mental 
health. 

Without  general  knowledge  in  mental  health  areas, 
we  statisticians  will  encounter  numerous  barriers  in 
communicating  with  mental  health  practitioners,  ad- 
ministrators, and  planners.  We  shall  be  unable  to  maxi- 
mize our  areas  of  expertise.  We  shall  have  difficulties 
conceptualizing  program  evaluation  models  and  visual- 
izing measurable  program  objective.  For  example,  we 
should  be  aware  of  the  shift  in  focus  of  mental  health 
treatment  delivery  simultaneously  toward  and  away 
from  private  practice.  For  most  patients  there  is  a de- 
creasing reliance  on  utilizing  private  practitioners 
(despite  sliding  scales  for  fee  payments)  and  more  on 
“public  funding,  public  program  development,  public 
institutions  as  the  main  vehicles  for  the  delivery  of 
care.”  (9)  We  should  also  see  the  move  toward  spe- 
cialization of  professional  services  as  well  as  the  move 
in  delivery-of-service  concepts  from  the  special  to  the 
general,  i.e.,  from  providing  services  to  defined  groups 
(such  as  the  mentally  ill)  to  providing  services  to 
almost  the  full  population.  Definitions  of  who  should 
receive  services  are  constantly  being  broadened  be- 
cause of  a variety  of  societal  and  other  influences.  An- 
other rather  obvious  shift  in  focus  is  that  paralleling 
public  health — concentrated  attention  upon  provid- 
ing overt  preventive  measures  in  addition  to  pallia- 
tive services.  That  is  in  recognition  of  the  fact  that 
control  of  disabilities  or  diseases  cannot  be  accom- 
plished merely  by  treating  the  disabled  or  diseased. 

The  statistician’s  role  in  program  planning  and  eval- 
uation is  that  of  a consultant  in  addition  to  that  of  a 
team  member.  As  a team  member,  the  statistician  may 


decide  that  considerations  relating  to  the  power  bases 
for  decisionmaking  are  germane.  As  a consultant,  he 
knows  that  although  he  makes  recommendations,  he 
does  not  necessarily  enter  into  actual  decisionmaking — 
not  even  in  all  subject  areas  specifically  within  his  ap- 
parent domain  of  expertise.  Therefore,  he  should  be 
acutely  and  constantly  aware  of  the  difference  between 
the  two  roles  even  through  his  role  relationships  may 
change  from  one  to  the  other  and  back  again  during 
and  between  meetings  of  groups  with  which  he  works. 

As  noted  earlier,  for  successful  participation  in  pro- 
gram planning  and  evaluation  “a  relatively  high  degree 
of  agreement  is  desirable  between  the  expectations  of 
the  researcher  and  those  of  the  agency.”  (10)  Ex- 
perience teaches  us  that  usually  practitioners,  adminis- 
trators, and  planners  are  interested  in  methods  which 
will  result  in  quick  action  and,  even  more,  in  favorable 
findings.  They  are  not  especially  interested  in  further- 
ing knowledge,  although  they  want  and  expect  the 
statisticians  to  be  knowledgeable  about  others’  works. 
On  the  other  hand,  introspection  will  reveal  that  we 
statisticians  may  strike  the  practitioners,  administrators, 
and  planners  as  being  overly  concerned  with  method- 
ology and  with  keeping  an  intact  research  design  (al- 
most an  impossibility  in  most  administrative  settings) . 
Thus,  in  studying  his  role  the  statistician  must  recognize 
that  in  emphasizing  the  theoretical  significance  of  his 
work  and  in  maintaining  rigorous  controls  with  respect 
to  research  methodology,  he  may  not  be  giving  proper 
perspective  to  eventual  utilization  of  findings  by  the 
agency  providing  client  services.  A middle  course  of 
action  becomes  mandatory  in  the  case  of  program  eval- 
uation. Compromises  must  be  made,  but  they  should  be 
carefully  considered  with  respect  not  only  to  meeting 
current  needs  but  also  to  anticipating  future  ramifica- 
tions. There  must  be  recognition  of  theoretical  and 
other  factors  which  cannot  be  compromised. 

The  primary  focus  of  this  presentation  has  been  to 
discuss  the  statistician’s  role  in  program  planning  and 
program  evaluation  from  the  viewpoint  of  enhancing 
effective  participation.  A review  of  bases  of  power 
in  decisionmaking  disclosed  that  the  power  base  most 
amenable  to  personal  intervention  and  most  related 
to  effective  participation  was  that  of  expertise  which 
I chose  to  describe  in  terms  of  motivation,  competence, 
technical  knowledge,  knowledge  of  the  field  of  activity, 
and  provision  of  consultative  services.  Development 
of  that  base  of  power  can  lead  to  development  in  other 
power  bases  and,  therefore,  to  subsequent  expansion 
in  decisionmaking  power  for  statisticians. 

Some  of  the  relationships  between  knowledge  and 
power  were  examined  by  Kan,-  Hyre  in  an  unpublished 
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paper  (11)  in  1968.  He  noted  the  importance  of  main- 
taining balance  among  problem  areas  and  being  aware 
of  the  integrative  nature  among  them.  In  that  context 
Hyre  asserted  axiomatically,  “When  too  great  an 
amount  of  time  is  spent  in  one  problem  area,  then  the 
time  necessary  for  solving  problems  in  other  areas  is 
too  severely  limited.”  In  transforming  that  assertion  to 
fit  our  current  considerations,  we  must  recognize  our 
need  to  expand  our  expertise  according  to  a planned 
approach.  We  cannot  focus  our  attention  completely 
on  technical  knowledge  areas.  We  cannot  become  ex- 
clusively engrossed  in  behavioral  dynamics  relating  to 
motivation  and  competence.  We  cannot  expend  in- 
ordinate amounts  of  time  reading  mental  health  jour- 
nals and  discoursing  the  mental  health  practitioners. 
Yet,  we  must  advance  in  many  areas  almost  simultane- 
ously. That  will  require  considerable  examination, 


evaluation  of  needs,  priority  setting,  planning,  and 
administrative  encouragement. 

Finally,  through  effective  provision  of  research  and 
statistical  consultation  (taking  into  account  the  ap- 
propriate utilization  of  expertise),  the  statistician  can 
contribute  much  toward  achieving  goals  he  shares  with 
the  administrator  and  the  planner-evaluator.  That  will 
lead  to  collective  approval  which,  as  Peter  Blau  (12) 
states,  “.  . . legitimates  leadership.  The  abilities  that 
enable  a person  to  make  major  contributions  to  the 
achievement  of  a group’s  goals  command  respect.  The 
respect  of  others  for  him  prompts  them  to  follow  his 
suggestions,  since  they  expect  to  benefit  from  doing 
so  . . . 

The  challenge  to  statisticians  is  clear.  Can  we  indi- 
vidually do  anything  about  it? 
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THE  PLANNER-EVALUATOR  ROLE  IN  A STATE  DEPARTMENT 

OF  MENTAL  HEALTH 

Leo  Levy,  Ph.  D.,  S.M.  hyg. 
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A formal  planning  mechanism  for  the  Department 
of  Mental  Health  was  institutionalized  in  the  State  of 
Illinois  in  1964.  At  that  time,  a division  was  set  up  in 
the  Illinois  Department  of  Mental  Health  called  the 
Division  of  Planning  and  Evaluation  Services.  This 
was  a new  division  which  was  set  up  initially  under 
the  impetus  of  Federal  funds  that  were  available 
for  a 2-year  planning  project  in  mental  health.  How- 
ever, at  the  outset,  the  intention  in  setting  up  the  divi- 
sion was  that  it  would  be  a permanent  division,  an 
integral  arm  of  mental  health  and  when  Federal  funds 
expired,  it  would  become  a regular  part  of  the  De- 
partment’s biennial  budget. 

Planning  was  defined  at  the  outset  as  a global  process 
which  encompassed  a wide  range  of  activities.  This 
included  conceptually  the  choice  and  establishment  of 
planning  objectives,  the  development  of  data  and  in- 
formation systems  to  supply  data  with  regard  to  the 
planning  objectives  and  examination  of  alternative 
paths  of  implementation  of  planned  objectives.  The 
evaluation  of  the  implementation  of  these  objectives 
and  following  this,  a corrective  replanning  cycle  which 
carries  us  once  again  through  the  steps  of  setting  ob- 
jectives, gathering  information,  implementing  pro- 
grams and  once  again  evaluating  them  was  incorpo- 
rated as  part  of  a systems  model  for  planning.1 

The  division’s  functions  at  the  outset  were  divided 
into  two  general  kinds  of  activities.  These  could  be 
referred  to  as  intramural  and  extramural  planning. 
Extramural  planning  took  the  form  of  setting  up  an 
independent  voluntary  planning  apparatus  to  do  long- 
range  planning  in  mental  health  for  the  State  of  Illinois 
which  would  be  representative  of  the  many  interested 
groups  in  the  State  who  inevitably  would  have  to  be  in- 
volved in  such  a process.  This  was  a first  priority  item 
because  Federal  monies  which  would  shortly  become 

__  t 

1 See  Division  of  Planning  and  Evaluation  Services  progress 
reports  for  years  1964,  1965,  1966,  and  1967,  Illinois  Depart- 
ment of  Mental  Health,  Division  of  Planning  and  Evaluation 
Services,  Chicago,  111. 


available  to  the  State  under  the  provisions  of  Public 
Law  88-164  required  that  a comprehensive  State  plan 
be  compiled.  This  long-range  programatic  plan  was 
seen  to  be  the  function  of  an  extramural  planning 
group  rather  than  a function  of  the  Department  of 
Mental  Health.  This  is  understandable  in  the  light  of 
modem  conceptions  of  mental  health  which  conceive 
of  mental  health  as  having  much  broader  scope  than 
the  question  of  the  treatment  of  the  mentally  ill  per  se. 
Planning  for  mental  health  was  conceived  of  as  a proc- 
ess which  would  involve  many  different  agencies  both 
public  and  private,  and  integrally  involve  such  areas 
as  education,  public  safety,  public  health,  vocational 
rehabilitation,  public  aid,  children  and  family  services, 
and  many  other  agencies  outside  the  purview  of  the 
Department  of  Mental  Health.  Consequently,  in  the 
fall  of  1964  the  Illinois  Mental  Health  Planning  Board 
was  established  with  certain  advisory  councils.  These 
councils  originally  numbered  six,  but  over  a period  of 
time  they  were  reduced  to  three.  The  three  which  re- 
mained were  the  Universities  Council,  Professional 
Societies  Council,  and  the  Council  of  State  and  Fed- 
eral Agencies.2  In  1967  the  planning  board  and  its  three 
advisory  councils  became  a statutory  body  charged  with 
long-range  planning  for  mental  health  and  mental  re- 
tardation in  the  State  of  Illinois. 

The  Planning  Division  of  the  Department  of  Mental 
Health  took  a very  active  role  in  setting  up  the  board 
and  councils  and  has  continuously  given  staff  support 
to  these  bodies  and  the  various  task  forces  originating 
from  these  bodies.  As  time  went  on,  the  administrative 
separation  of  the  planning  board  and  its  councils  from 
the  Department  of  Mental  Health  became  more  clear- 
cut  and  was  eventually  formalized  in  1967  when  the 
planning  board  achieved  statutory7  status  and  ap- 
pointed its  own  executive  secretary.  The  department’s 
Planning  Division  then  moved  to  a consultative  and 

- See  Progress  Reports  1,  2 and  3,  Illinois  Mental  Health 
Planning  Board,  Chicago,  111. 
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advisory  function  supplying  the  board  and  councils 
with  data  and  conceptual  expertise  on  their  request. 
The  Director  of  the  Division  of  Planning  and  Evalua- 
tion Services,  together  with  the  Director  of  the  Depart- 
ment of  Mental  Health  have  sat  continuously  as 
ex-officio  members  of  that  board. 

The  intramural  functions  of  the  Division  of  Planning 
and  Evaluation  Services  gradually  crystallized  into  four 
general  areas  of  responsibility  within  the  Department 
of  Mental  Health.  These  programatic  areas  may  be 
labeled:  (1)  Planning  support  and  consultation,  (2) 
program  evaluation,  (3)  social  research,  and  (4)  data 
collection  and  analysis. 

PLANNING  SUPPORT  AND  CONSULTATION 

This  is  a function  for  which  the  major  responsibility 
is  divided  among  the  division  director  and  two  assist- 
ant directors  who  have  been  trained  and  have  broad 
experience  in  the  area  of  community  organization.  Our 
general  objectives  here  are  the  development  of  com- 
munity-based comprehensive  community  mental  health 
programs  in  the  various  geographic  areas  of  our  State 
which  have  been  designated  as  zones.  Connected  with 
this  effort  are  the  development  of  the  State  mental 
health  and  mental  retardation  construction  plans, 
consultation  on  zone  and  subzonc  planning,  participa- 
tion in  and  preparation  of  Federal  grants  under  Public 
Laws  88-164  and  89-105  and  work  on  various  com- 
munity development  problems  such  as  the  passage  of 
mental  health  referenda  under  House  Bill  708. 
Through  the  Planning  Division,  staff  support  for  in- 
ternal departmental  planning  of  program  is  available 
upon  request.  This  staff  support  goes  freely  either 
upwards  or  downwards  in  the  department’s  bureau- 
cratic structure.  Thus,  the  Planning  Division  offers 
staff  support  to  the  director  and  his  special  assistants, 
but  also  to  the  zones,  the  zone  directors,  their  staffs 
and  the  directors  of  various  institutional  programs. 
One  major  recent  instance  of  the  planning  support 
function  was  the  recent  development  of  a plan  for 
the  delivery  of  services  in  the  Chicago  metropolitan 
area  initially  referred  to  as  Metrozone.3  This  rather 
complex  rational  plan  for  the  delivery  of  services  and 
the  allocation  of  departmental  resources  throughout 
the  Chicago  metropolitan  area  was  largely  the  out- 
come of  a collaborative  effort  among  four  people. 
These  were  the  director  and  assistant  director  of  the 
Planning  Division  assigned  to  the  Chicago  Metro- 

3 Metrozone,  the  Chicago  Area  Plan  for  Mental  Health 
Services,  October,  1968,  Illinois  Department  of  Mental 
Health,  Chicago,  111. 


politan  area,  a special  assistant  to  the  Department  di- 
rector, and  an  outside  university  consultant.  As  the 
basic  plan  took  form  and  as  Metrozone  moved  from 
planning  to  implementation,  all  four  of  these  principals 
dropped  out  of  the  picture  completely  and  a whole 
different  group  of  people  took  over  the  implementation 
under  a newly  appointed  Chicago  area  zone  director. 

Other  functions  associated  with  planning  support 
and  consultation  are,  as  mentioned  above,  the  staff 
support  to  the  Illinois  Mental  Health  Planning  Board 
and  its  affiliated  councils  and  a multitude  of  liaison 
functions  with  other  State  and  Federal  agencies,  co- 
ordinating mental  health  planning  with  parallel  plan- 
ning efforts  such  as  under  the  Model  Cities  Program 
and  under  the  Partnership  for  Health  Legislation,  Pub- 
lic Law  89-749. 

PROGRAM  EVALUATION 

This  aspect  of  the  division’s  program  functions  under 
the  leadership  of  the  chief  of  program  analysis  and 
evaluative  research  who  is  directly  responsible  to  the 
division  director.  This  person,  who  is  a statistician  with 
additional  background  in  the  behavioral  sciences,  is 
supported  by  a small  staff  and  in  collaboration  with 
the  data  collection  and  analysis  program,  develops 
models  for  evaluation  for  all  facets  of  the  department’s 
program.  The  program  evaluation  unit  functions 
mainly  on  the  basis  of  monitoring  statistical  reports 
furnished  through  the  data  collection  and  analysis  unit. 
The  program  evaluation  unit  monitors  operating  sta- 
tistics in  an  effort  to  provide  meaningful  commentary 
on  the  department’s  current  and  past  state  of  opera- 
tions. This  continuing  commentary  dealing  with  such 
topics  as  patient  movement  data — that  is,  admissions, 
length  of  stay,  discharge,  readmissions,  etc.,  significantly 
affects  both  the  administration  and  future  planning  of 
departmental  programs.  The  program  evaluation  unit 
also  functions  to  maintain  contact  and  consultation 
with  other  program  evaluation  units  situated  in  the 
zone  centers  and  State  hospitals,  and  has  provided  a 
manual  on  basic  principles  of  evaluative  research.4  This 
unit  has  not,  up  to  the  present  time,  attempted  to  do 
controlled  evaluation  studies.  This  level  of  evaluation 
has  been  delegated  to  the  zone  staffs,  each  of  whom 
has  the  potentiality  of  developing  a local  program  eval- 
uation unit.  Thus,  the  Planning  Division’s  evaluative 
function,  up  to  the  present  time,  has  been  defined 
mainly  in  the  area  of  epidemiological  surveillance. 

4 Manual  for  Evaluation  of  Mental  Health  Programs,  May 
1,  1966,  Illinois  Department  of  Mental  Health,  Division  of 
Planning  and  Evaluation  Services,  Chicago,  111. 
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SOCIAL  RESEARCH 

This  unit,  principally  staffed  by  three  behavioral 
scientists  and  supported  by  research  analysts  and 
clerical  staff,  performs  epidemiological  and  demo- 
graphic research  principally  related  to  the  general  area 
of  program  development.  Each  of  the  behavioral  sci- 
entists is  directly  responsible  to  the  division  director 
in  terms  of  his  ongoing  research  activities.  Much  valu- 
able output  has  already  come  from  this  unit,  studying 
such  topics  as  the  structure  and  organization  of  our 
State  hospitals,  analysis  of  community  variables  relat- 
ing to  admission  to  State  hospitals,  relation  of  popula- 
tion density  to  the  occurrence  of  psychiatric  casualties, 
etc.  The  behavioral  scientists,  while  encouraged  to 
work  collaboratively  with  each  other,  with  the  division 
director  and  with  other  members  of  the  Planning 
Division  staff,  are  allowed  considerable  autonomy  in 
terms  of  selecting  research  projects  on  which  they  wish 
to  work.  Their  research  is  of  an  applied  nature  and  in 
the  final  analysis,  must  be  capable  of  demonstrating 
some  link  with  program  development  and  program 
planning.  However,  within  these  very  broad  con- 
straints, they  have  considerable  freedom  to  select  prob- 
lems which  interest  them  and  which  suit  their 
particular  area  of  expertise. 

DATA  COLLECTION  AND  ANALYSIS 

The  Planning  Division,  although  established  de 
novo,  was  nonetheless  not  without  responsibility.  One 
considerable  responsibility  was  to  take  over  the  direc- 
tion of  the  statistical  research  unit  which  had  been 
previously  in  the  department’s  administrative  section. 
This  unit  was  concerned  primarily  with  supplying  the 
NIMH  biometry  section  with  annual  patient  move- 
ment statistics  and  preparing  various  routine  statistical 
reports.  At  the  time  this  organizational  change  was 
made,  the  section  had  become  almost  moribund,  de- 
livering its  reports  anywhere  between  one  and  two 
years  late.  Obviously,  no  immediate  improvement 
could  be  accomplished  since  it  is  quite  difficult  to  over- 
come the  degenerative  process  once  it  sets  in,  in  a 
statistical  reporting  unit.  What  happens  is  that  a gen- 
eral malaise  overtakes  the  unit  as  well  as  the  persons 
providing  the  information  and  as  a result,  the  data 
collected  becomes  increasingly  degraded  both  with 
respect  to  quality  and  timeliness. 

About  the  same  time  that  the  Planning  Division  was 
established,  or  perhaps  a year  earlier,  the  department 
decided  to  transfer  its  data-processing  operations  from 
unit  record  equipment  to  a tape  computer  system.  The 


decision  was  made  to  obtain  a Burroughs  B-280  com- 
puter system.  As  you  may  well  imagine,  this  created 
considerable  pain  both  organizationally  and  to  the 
employees  since  the  first  and  most  serious  problem 
which  resulted  was  late  payrolls  and  late  billing  of 
patients.  Therefore  the  need  to  computerize  the  pa- 
tient statistical  operation  became  of  secondary  im- 
portance so  that  patient  statistics  continued  to  be 
maintained  on  a manual  and  punched  card  basis.  This 
meant  that  analysis  of  the  statistical  data  could  only 
be  made  at  the  end  of  the  fiscal  year  when  the  files 
could  be  frozen  for  analysis.  Thus,  one  initial  aim  of 
the  Planning  Division  was  to  computerize  the  patient 
recordkeeping  activity  in  order  to  provide  more  timely 
data.  This,  however,  could  not  be  done  without  sacri- 
fice. The  sacrifice  that  was  made  was  to  also  take  on 
the  responsibility  of  running  the  rapidly  deteriorating 
data  processing  function.  The  solution  was  to  combine 
the  statistical  research  unit  and  the  data  processing 
section  into  a new  section  with  the  name  of  the  Man- 
agement Information  Section.  In  retrospect,  from  the 
point  of  view  of  improving  statistical  reporting,  this 
now  seems  to  have  been  a monumental  mistake.  We 
did,  in  fact,  substantially  improve  the  functioning  of 
all  management  information  systems,  but  since  they 
always  had  a higher  priority,  we  neglected  even  fur- 
ther the  statistical  reporting  systems.  There  is  perhaps 
a useful  moral  here,  namely,  “Do  your  own  Thing.” 
An  additional  organizational  moral  which  follows  from 
this  is  that  one  should  avoid  deploying  limited  re- 
sources too  widely  or  nothing  useful  may  be  accom- 
plished. The  slogan  here  is  “Be  a fanatic”  if  you  want 
to  accomplish  something  substantial.  It  would  appear 
that  if  we  had  concentrated  all  our  efforts  on  improv- 
ing the  statistical  research  function  and  let  the  other 
management  information  needs  be  met  as  best  they 
could  by  others,  we  would  have  done  what  we  knew 
best  more  effectively  and  in  the  long  run  achieved 
much  more.  That  is  to  say,  we  would  have  been  more 
like  NASA  than  NIMH — more  concerned  with 
achieving  one  big  thing  well  regardless  of  duplication, 
expense  and  conflict  than  with  many  small  things  in- 
adequately. Thus  in  a real  sense,  we  are  still  using 
first  generation  concepts  (patient  movement  statis- 
tics) with  second  generation  programs  (COBOL)  and 
file  concepts,  on  a third  generation  computer  (IBM 
360/40) . 

Like  many  people  in  government  service,  we  were 
also  bemused  by  the  siren  song  of  program  budget- 
ing. We  devoted  a considerable  amount  of  our 
limited  resources  and  time  to  developing  a program 
budget  chart  of  accounts  and  coding  employees  (or  po- 
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sitions)  and  other  items  of  expenditure  by  this  new 
coding  scheme.  In  addition,  we  implemented  it  as  an 
information  and  control  system.  Unfortunately,  it  has 
never  worked  adequately  for  a very  simple  reason;  no- 
body ever  took  it  seriously  enough  to  make  financial 
decisions  on  the  basis  of  this  information.  So  the  data 
has  become  increasingly  degraded.  In  addition,  as 
with  many  planning  efforts,  management  was  in  a 
hurry  to  implement  the  system  so  that  we  approached 
the  problem  from  the  wrong  end.  Rather  than  making 
a detailed  analysis  of  the  actual  activities  within  the 
facilities,  we  started  with  the  programs  and  on  the 
basis  of  logical  analysis  defined,  the  activities.  This 
resulted  in  a failure  to  involve  adequately  the  institu- 
tional staff  in  the  process  of  defining  their  programs 
and  resulted  in  a very  logical,  complete  and  generally 
useless  chart  of  program  accounts.  In  addition,  it 
turned  out  that  this  was  not  really  what  was  wanted. 
What  was  actually  required  was  a cost  accounting 
scheme  showing  cost  responsibility  centers  and  a 
statistical  reporting  system  on  patient  and  staff  serv- 
ices that  could  relate  performance  to  budget.  Since  to 
do  this  latter  is  extremely  expensive  and  complicated, 
we  did  the  simpler  and  cheaper  thing  which  turned 
out  to  be  unsatisfactory.  There  is  another  moral  here. 
The  advantages  of  quick  gimmicky  solutions  over 
honest  toil  are  very  short-run  advantages.  We  did, 
however,  computerize  the  statistical  patient  informa- 
tion system  and  designed  an  integrated  payroll  per- 
sonnel reporting  and  control  system. 

From  the  above  brief  discussion,  it  should  be  evident 
that  the  Planning  Division  is  staffed  by  persons  with 
diversified  talents.  This  is  intentional  and  in  keeping 
with  the  stance  which  the  division  has  tried  to  develop, 
of  maintaining  a generalist  function.  We  like  to  think 
of  ourselves  as  a general  resource,  for  the  department 
and  for  the  planning  board,  for  consultation  and  staff 
support  in  a wide  variety  of  areas  which  pertain  to 
the  planning  and  evaluation  process.  There  are  several 
other  comments  which  ought  to  be  made  in  conclusion 
about  the  Planning  Division  and  its  relation  to  the  rest 
of  the  Department  of  Mental  Health.  First,  one  ought 
to  note  that  we  are  a staff  division,  which  means  that 
we  have  no  line  authority  or  responsibility  for  the  actual 
service  delivery  system.  It  is  not  even  mandatory  that 
we  be  involved  in  the  planning  of  any  specific  service 
delivery  system.  The  Illinois  Department  of  Mental 
Health  is  a highly  decentralized  department.  Each  zone 
is  a microcosm  unto  itself.  The  zone  director  is  a local 
district  director  of  mental  health.  There  are  very  few 
points  of  mandatory  interaction  among  the  zone  di- 
rectors and  the  Planning  Division.  One  such  point  of 


mandatory  interaction  is  on  the  topic  of  the  content  of 
a statewide  data  collection  system  on  patients.  Most 
of  the  contacts,  however,  are  voluntary  and  most  often 
made  at  the  request  of  the  zone  director.  The  director  of 
the  department,  however,  has  made  extensive  use  of 
the  Planning  Division  in  attempting  to  formulate  poli- 
cies and  answers  to  problems  which  have  a statewide 
implication.  It  is  natural  that  he  should  do  so  because 
each  zone  director  and  each  zone  staff  has  an  orienta- 
tion only  towards  a particular  portion  of  the  State. 
The  Planning  Division  has  no  particular  vested  interest 
in  any  one  zone. 

Secondly,  it  should  be  noted  that  the  planning  di- 
vision has  been  consciously  constructed  as  a division 
which  will  change  its  structure  and  function  rather  con- 
tinuously in  accommodating  to  changes  in  depart- 
mental functioning.  One  example  of  this  has  to  do  with 
the  role  of  the  assistant  division  directors.  These  men, 
as  noted  previously,  are  highly  trained  in  community 
organization.  When  the  Planning  Division  first  came 
into  existence  in  1964,  none  of  the  zone  staffs  had  com- 
munity organization  people  to  work  in  the  communities. 
Thus,  ajt  the  beginning,  these  two  men  very  often 
worked  as  community  organizers  in  zones,  working 
towards  implementing  various  departmental  objectives. 
This  function  has  gradually  changed  because  the  zone 
staffs  have  employed  their  own  community  organiza- 
tion personnel.  Today,  these  men  function  much  more 
in  the  role  of  planning  and  community  organization 
consultants  and  do  very  little  direct  community  orga- 
nization work  for  the  department.  Another  example  of 
the  transitional  nature  of  the  division’s  role  has  to  do 
with  management  information.  This  function,  as  indi- 
cated before,  was  gradually  appropriated  by  the  divi- 
sion out  of  logical  necessity  over  the  first  year  and  one- 
half  of  its  existence.  However,  it  has  never  been  con- 
sidered necessary  or  even  appropriate  that  the  Planning 
Division  actually  run  the  management  information 
operation.  With  regard  to  the  management  informa- 
tion section,  the  Planning  Division  is  a consumer  like 
many  other  people  in  the  department.  The  Planning 
Division  has  helped  to  develop  the  management  infor- 
mation function  to  a point,  at  the  present  time,  where 
it  functions  pretty  autonomously  as  a separate  unit.  It 
is  quite  conceivable  in  the  near  future  that  this  section 
may  achieve  separate  divisional  status  or  move  to  some 
other  structural  position  within  the  department. 

As  a third  and  final  point,  it  should  be  noted  that 
if  the  Planning  Division  has  any  content  function  as 
such,  it  might  be  defined  as  an  orientation  towards  the 
development  of  comprehensive  community  mental 
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health  programs  on  a highly  decentralized  level.  Thus, 
the  division  quite  naturally  takes  an  interest  in  the 
proper  implementation  of  such  legislation  as  Public 
Laws  88-164  and  89-105  and  other  related  legislation 
which  enables  the  establishment  of  local  comprehen- 
sive community  mental  health  centers. 

The  Division  of  Planning  and  Evaluation  Services  of 
the  Illinois  Department  of  Mental  Health  has,  in  a 


sense,  been  an  interesting  and  instructive  experiment. 
The  experiment  is  now  entering  its  sixth  year.  The  issue 
is  no  longer  whether  planning  and  evaluation  have  a 
function  in  such  a department,  but  rather  how  one 
optimally  fits  such  a function  to  a large  and  complex 
agency.  Our  way  as  outlined  in  this  paper  is  but  one  of 
several  conceptual  models  which  have  developed  in 
the  country. 
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